外贸技术家园ElasticSearch是一个基于Lucene的搜索服务器。它提供了一个分布式多用户能力的全文搜索引擎，基于RESTFul web接口。ElasticSearch是用Java开发的，并作为Apache许可条款下的开放源码发布，是当前流行的企业级搜索引擎。ElasticSearch常用于全文检索，结构化检索，数据分析等。
下面，我们以ElasticSearch接管Linux日志（/var/log/xxx.log）为例，详细介绍如何进行配置与部署。
总体架构图
一，准备工作
1，CVM及ElasticSearch
在腾讯云帐号下，申请一台CVM（Linux操作系统）、一个ElasticSearch集群（后面简称ES），使用最简配置即可；申请的CVM和ES，必须在同一个VPC的同一个子网下。
CVM详情信息
ElasticSearch详情信息
2，Filebeat工具
为了将Linux日志提取到ES中，我们需要使用Filebeat工具。Filebeat是一个日志文件托运工具，在你的服务器上安装客户端后，Filebeat会监控日志目录或者指定的日志文件，追踪读取这些文件（追踪文件的变化，不停的读），并且转发这些信息到ElasticSearch或者logstarsh中存放。当你开启Filebeat程序的时候，它会启动一个或多个探测器（prospectors）去检测你指定的日志目录或文件，对于探测器找出的每一个日志文件，Filebeat启动收割进程（harvester），每一个收割进程读取一个日志文件的新内容，并发送这些新的日志数据到处理程序（spooler），处理程序会集合这些事件，最后Filebeat会发送集合的数据到你指定的地点。
官网简介：https://www.elastic.co/products/beats/filebeat
二，操作步骤
1，Filebeat下载与安装
首先，登录待接管日志的CVM，在CVM上下载Filebeat工具：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
[root@VM_3_7_centos ~]<span class="hljs-comment"># cd /opt/</span> [root@VM_3_7_centos opt]<span class="hljs-comment"># ll</span> total 4 drwxr-xr-x. 2 root root 4096 Sep 7 2017 rh [root@VM_3_7_centos opt]<span class="hljs-comment"># wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.2-x86_64.rpm</span> --2018-12-10 20:24:26-- https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.2-x86_64.rpm Resolving artifacts.elastic.co (artifacts.elastic.co)... 107.21.202.15, 107.21.127.184, 54.225.214.74, ... Connecting to artifacts.elastic.co (artifacts.elastic.co)|107.21.202.15|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 12697788 (12M) [binary/octet-stream] Saving to: ‘filebeat-6.2.2-x86_64.rpm’ 100%[=================================================================================================>] 12,697,788 160KB/s <span class="hljs-keyword">in</span> 1m 41s 2018-12-10 20:26:08 (123 KB/s) - ‘filebeat-6.2.2-x86_64.rpm’ saved [12697788/12697788] |
然后，进行安装filebeat：
|
1 2 3 4 5 6 |
[root@VM_3_7_centos opt]<span class="hljs-comment"># rpm -vi filebeat-6.2.2-x86_64.rpm</span> warning: filebeat-6.2.2-x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY Preparing packages... filebeat-6.2.2-1.x86_64 [root@VM_3_7_centos opt]<span class="hljs-comment">#</span> |
至此，Filebeat安装完成。
2，Filebeat配置
进入Filebeat配置文件目录：/etc/filebeat/
|
1 2 3 4 5 6 7 8 9 |
[root@VM_3_7_centos opt]<span class="hljs-comment"># cd /etc/filebeat/</span> [root@VM_3_7_centos filebeat]<span class="hljs-comment"># ll</span> total 108 -rw-r--r-- 1 root root 44384 Feb 17 2018 fields.yml -rw-r----- 1 root root 52193 Feb 17 2018 filebeat.reference.yml -rw------- 1 root root 7264 Feb 17 2018 filebeat.yml drwxr-xr-x 2 root root 4096 Dec 10 20:35 modules.d [root@VM_3_7_centos filebeat]<span class="hljs-comment">#</span> |
其中，filebeat.yml就是我们需要修改的配置文件。建议修改配置前，先备份此文件。
然后，确认需要对接ElasticSearch的Linux的日志目录，我们以下图（/var/log/secure）为例。
/var/log/secure日志文件
使用vim打开/etc/filebeat/filebeat.yml文件，修改其中的：
1）Filebeat prospectors类目中，enable默认为false，我们要改为true
2）paths，默认为/var/log/*.log，我们要改为待接管的日志路径：/var/log/secure
3）Outputs类目中，有ElasticSearchoutput配置，其中hosts默认为”localhost:9200″，需要我们手工修改为上面申请的ES子网地址和端口，即**”10.0.3.8:9200″**。
修改好上述内容后，保存退出。
修改好的配置文件全文如下：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 |
[root@VM_3_7_centos /]<span class="hljs-comment"># vim /etc/filebeat/filebeat.yml</span> [root@VM_3_7_centos /]<span class="hljs-comment"># cat /etc/filebeat/filebeat.yml</span> <span class="hljs-comment">###################### Filebeat Configuration Example #########################</span> <span class="hljs-comment"># This file is an example configuration file highlighting only the most common</span> <span class="hljs-comment"># options. The filebeat.reference.yml file from the same directory contains all the</span> <span class="hljs-comment"># supported options with more comments. You can use it as a reference.</span> <span class="hljs-comment">#</span> <span class="hljs-comment"># You can find the full configuration reference here:</span> <span class="hljs-comment"># https://www.elastic.co/guide/en/beats/filebeat/index.html</span> <span class="hljs-comment"># For more available modules and options, please see the filebeat.reference.yml sample</span> <span class="hljs-comment"># configuration file.</span> <span class="hljs-comment">#=========================== Filebeat prospectors =============================</span> filebeat.prospectors: <span class="hljs-comment"># Each - is a prospector. Most options can be set at the prospector level, so</span> <span class="hljs-comment"># you can use different prospectors for various configurations.</span> <span class="hljs-comment"># Below are the prospector specific configurations.</span> - <span class="hljs-built_in">type</span>: <span class="hljs-built_in">log</span> <span class="hljs-comment"># Change to true to enable this prospector configuration.</span> enabled: <span class="hljs-literal">true</span> <span class="hljs-comment"># Paths that should be crawled and fetched. Glob based paths.</span> paths: - /var/<span class="hljs-built_in">log</span>/secure <span class="hljs-comment">#- c:\programdata\elasticsearch\logs\*</span> <span class="hljs-comment"># Exclude lines. A list of regular expressions to match. It drops the lines that are</span> <span class="hljs-comment"># matching any regular expression from the list.</span> <span class="hljs-comment">#exclude_lines: ['^DBG']</span> <span class="hljs-comment"># Include lines. A list of regular expressions to match. It exports the lines that are</span> <span class="hljs-comment"># matching any regular expression from the list.</span> <span class="hljs-comment">#include_lines: ['^ERR', '^WARN']</span> <span class="hljs-comment"># Exclude files. A list of regular expressions to match. Filebeat drops the files that</span> <span class="hljs-comment"># are matching any regular expression from the list. By default, no files are dropped.</span> <span class="hljs-comment">#exclude_files: ['.gz$']</span> <span class="hljs-comment"># Optional additional fields. These fields can be freely picked</span> <span class="hljs-comment"># to add additional information to the crawled log files for filtering</span> <span class="hljs-comment">#fields:</span> <span class="hljs-comment"># level: debug</span> <span class="hljs-comment"># review: 1</span> <span class="hljs-comment">### Multiline options</span> <span class="hljs-comment"># Mutiline can be used for log messages spanning multiple lines. This is common</span> <span class="hljs-comment"># for Java Stack Traces or C-Line Continuation</span> <span class="hljs-comment"># The regexp Pattern that has to be matched. The example pattern matches all lines starting with [</span> <span class="hljs-comment">#multiline.pattern: ^\[</span> <span class="hljs-comment"># Defines if the pattern set under pattern should be negated or not. Default is false.</span> <span class="hljs-comment">#multiline.negate: false</span> <span class="hljs-comment"># Match can be set to "after" or "before". It is used to define if lines should be append to a pattern</span> <span class="hljs-comment"># that was (not) matched before or after or as long as a pattern is not matched based on negate.</span> <span class="hljs-comment"># Note: After is the equivalent to previous and before is the equivalent to to next in Logstash</span> <span class="hljs-comment">#multiline.match: after</span> <span class="hljs-comment">#============================= Filebeat modules ===============================</span> filebeat.config.modules: <span class="hljs-comment"># Glob pattern for configuration loading</span> path: <span class="hljs-variable">${path.config}</span>/modules.d/*.yml <span class="hljs-comment"># Set to true to enable config reloading</span> reload.enabled: <span class="hljs-literal">false</span> <span class="hljs-comment"># Period on which files under path should be checked for changes</span> <span class="hljs-comment">#reload.period: 10s</span> <span class="hljs-comment">#==================== Elasticsearch template setting ==========================</span> setup.template.settings: index.number_of_shards: 3 <span class="hljs-comment">#index.codec: best_compression</span> <span class="hljs-comment">#_source.enabled: false</span> <span class="hljs-comment">#================================ General =====================================</span> <span class="hljs-comment"># The name of the shipper that publishes the network data. It can be used to group</span> <span class="hljs-comment"># all the transactions sent by a single shipper in the web interface.</span> <span class="hljs-comment">#name:</span> <span class="hljs-comment"># The tags of the shipper are included in their own field with each</span> <span class="hljs-comment"># transaction published.</span> <span class="hljs-comment">#tags: ["service-X", "web-tier"]</span> <span class="hljs-comment"># Optional fields that you can specify to add additional information to the</span> <span class="hljs-comment"># output.</span> <span class="hljs-comment">#fields:</span> <span class="hljs-comment"># env: staging</span> <span class="hljs-comment">#============================== Dashboards =====================================</span> <span class="hljs-comment"># These settings control loading the sample dashboards to the Kibana index. Loading</span> <span class="hljs-comment"># the dashboards is disabled by default and can be enabled either by setting the</span> <span class="hljs-comment"># options here, or by using the `-setup` CLI flag or the `setup` command.</span> <span class="hljs-comment">#setup.dashboards.enabled: false</span> <span class="hljs-comment"># The URL from where to download the dashboards archive. By default this URL</span> <span class="hljs-comment"># has a value which is computed based on the Beat name and version. For released</span> <span class="hljs-comment"># versions, this URL points to the dashboard archive on the artifacts.elastic.co</span> <span class="hljs-comment"># website.</span> <span class="hljs-comment">#setup.dashboards.url:</span> <span class="hljs-comment">#============================== Kibana =====================================</span> <span class="hljs-comment"># Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.</span> <span class="hljs-comment"># This requires a Kibana endpoint configuration.</span> setup.kibana: <span class="hljs-comment"># Kibana Host</span> <span class="hljs-comment"># Scheme and port can be left out and will be set to the default (http and 5601)</span> <span class="hljs-comment"># In case you specify and additional path, the scheme is required: http://localhost:5601/path</span> <span class="hljs-comment"># IPv6 addresses should always be defined as: https://[2001:db8::1]:5601</span> <span class="hljs-comment">#host: "localhost:5601"</span> <span class="hljs-comment">#============================= Elastic Cloud ==================================</span> <span class="hljs-comment"># These settings simplify using filebeat with the Elastic Cloud (https://cloud.elastic.co/).</span> <span class="hljs-comment"># The cloud.id setting overwrites the `output.elasticsearch.hosts` and</span> <span class="hljs-comment"># `setup.kibana.host` options.</span> <span class="hljs-comment"># You can find the `cloud.id` in the Elastic Cloud web UI.</span> <span class="hljs-comment">#cloud.id:</span> <span class="hljs-comment"># The cloud.auth setting overwrites the `output.elasticsearch.username` and</span> <span class="hljs-comment"># `output.elasticsearch.password` settings. The format is `<user>:<pass>`.</span> <span class="hljs-comment">#cloud.auth:</span> <span class="hljs-comment">#================================ Outputs =====================================</span> <span class="hljs-comment"># Configure what output to use when sending the data collected by the beat.</span> <span class="hljs-comment">#-------------------------- Elasticsearch output ------------------------------</span> output.elasticsearch: <span class="hljs-comment"># Array of hosts to connect to.</span> hosts: [<span class="hljs-string">"10.0.3.8:9200"</span>] <span class="hljs-comment"># Optional protocol and basic auth credentials.</span> <span class="hljs-comment">#protocol: "https"</span> <span class="hljs-comment">#username: "elastic"</span> <span class="hljs-comment">#password: "changeme"</span> <span class="hljs-comment">#----------------------------- Logstash output --------------------------------</span> <span class="hljs-comment">#output.logstash:</span> <span class="hljs-comment"># The Logstash hosts</span> <span class="hljs-comment">#hosts: ["localhost:5044"]</span> <span class="hljs-comment"># Optional SSL. By default is off.</span> <span class="hljs-comment"># List of root certificates for HTTPS server verifications</span> <span class="hljs-comment">#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]</span> <span class="hljs-comment"># Certificate for SSL client authentication</span> <span class="hljs-comment">#ssl.certificate: "/etc/pki/client/cert.pem"</span> <span class="hljs-comment"># Client Certificate Key</span> <span class="hljs-comment">#ssl.key: "/etc/pki/client/cert.key"</span> <span class="hljs-comment">#================================ Logging =====================================</span> <span class="hljs-comment"># Sets log level. The default log level is info.</span> <span class="hljs-comment"># Available log levels are: error, warning, info, debug</span> <span class="hljs-comment">#logging.level: debug</span> <span class="hljs-comment"># At debug level, you can selectively enable logging only for some components.</span> <span class="hljs-comment"># To enable all selectors use ["*"]. Examples of other selectors are "beat",</span> <span class="hljs-comment"># "publish", "service".</span> <span class="hljs-comment">#logging.selectors: ["*"]</span> <span class="hljs-comment">#============================== Xpack Monitoring ===============================</span> <span class="hljs-comment"># filebeat can export internal metrics to a central Elasticsearch monitoring</span> <span class="hljs-comment"># cluster. This requires xpack monitoring to be enabled in Elasticsearch. The</span> <span class="hljs-comment"># reporting is disabled by default.</span> <span class="hljs-comment"># Set to true to enable the monitoring reporter.</span> <span class="hljs-comment">#xpack.monitoring.enabled: false</span> <span class="hljs-comment"># Uncomment to send the metrics to Elasticsearch. Most settings from the</span> <span class="hljs-comment"># Elasticsearch output are accepted here as well. Any setting that is not set is</span> <span class="hljs-comment"># automatically inherited from the Elasticsearch output configuration, so if you</span> <span class="hljs-comment"># have the Elasticsearch output configured, you can simply uncomment the</span> <span class="hljs-comment"># following line.</span> <span class="hljs-comment">#xpack.monitoring.elasticsearch:</span> [root@VM_3_7_centos /]<span class="hljs-comment"># </span> |
执行下列命令启动filebeat
|
1 2 3 4 |
[root@VM_3_7_centos /]# sudo /etc/init.d/filebeat start Starting filebeat (via systemctl): [ OK ] [root@VM_3_7_centos /]# |
3，Kibana配置
进入ElasticSearch对应的Kibana管理页，如下图。
首次访问Kibana默认会显示管理页
首次登陆，会默认进入Management页面，我们需要将Index pattern内容修改为：filebeat-*，然后页面会自动填充**Time Filter field name，**不需手动设置，直接点击Create即可。点击Create后，页面需要一定时间来加载配置和数据，请稍等。如下图：
将Index pattern内容修改为：filebeat-*，然后点击Create
至此，CVM上，/var/log/secure日志文件，已对接到ElasticSearch中，历史日志可以通过Kibana进行查询，最新产生的日志也会实时同步到Kibana中。
三，实战效果
日志接管已完成配置，如何使用呢？
如下图：
在Index Patterns中可以看到我们配置过的filebeat-*
点击Discover，即可看到secure中的所有日志，页面上方的搜索框中输入关键字，即可完成日志的检索。如下图（点击图片，可查看高清大图）：
使用Kibana进行日志检索
实际上，检索只是Kibana提供的诸多功能之一，还有其他功能如可视化、分词检索等，还有待后续研究。
