MySQL日志收集之Filebeat和Logstsh的一键安装配置ELK架构

作者: 东山絮柳仔

关于ELK是什么、做什么用,们不在此讨论。本文重点在如何实现快速方便地安装logstash和filebeat组件,特别是在近千台DB Server的环境下(为了安全保守,公司DB Server 目前尚未部署saltstack一类的管控软件)。在尽可能标准化的条件下,希望可以实现一键化安装。下面是们功能实现的一些尝试,们把手动一步步操作打包提炼到一个sh文档中,安装部署时只要执行sh文件即可。部署安装logstash和filebeat组件由原来的10分钟缩减到目前的1分钟左右,并且减少了因手动部署带来的误操作。 1.logstash和filebeat安装包所在指定路径下 logstash的安装包logstash-7.6.0.zip所在路径 /data/logstash/logstash-7.6.0.zip

filebeat的安装包filebeat-7.4.2-linux-x86_64.tar.gz所在路径

2.上传经过标准化的程序配置文件

上传程序的配置文件filebeat.service、filebeat.yml、logstash.conf、startup.options到指定位置,这些文件是格式化后的,不是解压的默认文件,目的是方便替换安装。

如何想直接使用disposelogcollectot.sh文件,上传的路径一定要是/tmp/ 3.编写一键安装的可执行文件disposelogcollectot.sh

!/bin/bash
 The version is defined V.001
 Version   ModifyTime                ModifyBy              Desc
 Ver001    2018-03-25            Carson.Xu             Create the Scripts File
 Desc: This file is used to despose filebeat \ logstash in order to  collect slow log and error log from mysqld.
 step 1 判断 需要上传的文件是否已上传
cd /tmp/
if [ -f "filebeat.service" -a -f "filebeat.yml" -a -f "logstash.conf" -a -f "startup.options" ]
then
    echo 'step 1 安装过程需要的文件已到位,上传文件项检查通过....'
else
    echo "step 1 安装过程中需要的filebeat.service、 filebeat.yml、 logstash.conf、 startup.options,没有到位,不能继续安装,安装进程退出!!!"
    exit
fi
 step 2 解压指定文件
cd /data/logstash/
unzip logstash-7.6.0.zip
echo 'step 2 解压logstash项工作完成....'
sleep 3
 step 3 删除解压后的指定文件
cd logstash-7.6.0/config/
rm -rf startup.options
echo 'step 3 删除解压后的指定文件startup.options工作完成....'
sleep 3
 step 4 转移上传的文件
mv /tmp/logstash.conf /tmp/startup.options -t /data/logstash/logstash-7.6.0/config/
echo 'step 4 转移文件logstash.conf的工作完成....'
sleep 2
 step 5 修改log上传的ES 索引[必做 建议用业务名称替换,例如qq/weixin/rewu]
read  -p "请输入业务名称:" product
echo -e "\n"
echo "用户名为:$product"
sed -i "s/qqweixinface/$product/" /data/logstash/logstash-7.6.0/config/logstash.conf
echo 'step 5 删除解压后的指定文件startup.options工作完成....'
sleep 2
 step 6 安装logstash 服务
/data/logstash/logstash-7.6.0/bin/system-install
echo 'step 6 安装logstash 服务工作完成....'
sleep 3
 step 7 解压缩filebeat文件
cd /data/filebeat/
tar -zxvf filebeat-7.4.2-linux-x86_64.tar.gz
echo 'step 7 解压缩filebeat文件工作完成....'
sleep 3
 step 8 转移上传的filebeat.yml,允许覆盖掉生成默认配置文件
rm -rf /data/filebeat/filebeat-7.4.2-linux-x86_64/filebeat.yml
mv /tmp/filebeat.yml /data/filebeat/filebeat-7.4.2-linux-x86_64/
echo 'step 8 转移上传的filebeat.yml,允许覆盖掉生成默认配置文件工作完成....'
sleep 2
 step 9 权限调整
cd /data/filebeat/filebeat-7.4.2-linux-x86_64
chown -R root:root filebeat.yml
chmod 600 filebeat.yml
echo 'step 9 调整filebeat文件权限的工作完成....'
sleep 2
 step 10 获取 Server IP
ip=$(ip a|awk -F "inet|/"  '/inet.*brd/ '|head -n 1)
serverid=$(echo $ip) 去除左右空格
echo $serverid
echo 'step 10 获取Server IP的工作完成....'
echo '获取Server IP的为:' $serverid
sleep 1
 step 11 调整host配置[必做 IP替换]
sed -i "s/119.119.119.119/$serverid/" /data/filebeat/filebeat-7.4.2-linux-x86_64/filebeat.yml
echo 'step 11 替换配置文件中的Server IP工作完成....'
sleep 1
 step 12 转移服务文件
mv /tmp/filebeat.service /etc/systemd/system/
echo 'step 12 将filebeat服务的文件移动到指定位置的工作完成....'
sleep 1
 step 13 服务设置及启动
systemctl enable logstash.service
systemctl enable filebeat.service
echo 'step 13 将服务设置为自启动的工作完成....'
systemctl start logstash.service
sleep 20
systemctl start filebeat.service
sleep 10
 step 14 检查服务是否已正常启动
logstashservice_check_result=`systemctl status logstash.service | grep "active (running)"| wc -l`
if [ "$logstashservice_check_result" == "1" ]
then
    echo 'step 14 检查logstash.service已启动....'
else
    echo "step 14 检查logstash.service未正常启动....,安装进程退出!!!"
    exit
fi
sleep 3
filebeatservice_check_result=`systemctl status filebeat.service | grep "active (running)"| wc -l`
if [ "$filebeatservice_check_result" == "1" ]
then
    echo 'step 14 检查filebeat.service已启动....'
else
    echo "step 14 检查filebeat.service未正常启动....,安装进程退出!!!"
    exit
fi
 step 15 安装过程结束
echo 'step 15 安装过程结束'

4.执行

chmod 755 disposelogcollectot.sh
sh -x disposelogcollectot.sh

5.附录

在附录这一部分,介绍刚刚上传的文件–filebeat.service、filebeat.yml、logstash.conf、startup.options 5.1 文件startup.options

这一个文件主要描述了logstash程序的启动配置

 These settings are ONLY used by $LS_HOME/bin/system-install to create a custom
 startup script for Logstash and is not used by Logstash itself. It should
 automagically use the init system (systemd, upstart, sysv, etc.) that your
 Linux distribution uses.
 After changing anything here, you need to re-run $LS_HOME/bin/system-install
 as root to push the changes to the init script.
 Override Java location
JAVACMD=/usr/bin/java
 Set a home directory
LS_HOME=/data/logstash/logstash-7.6.0
 logstash settings directory, the path which contains logstash.yml
LS_SETTINGS_DIR=/data/logstash/logstash-7.6.0/config
 Arguments to pass to logstash
LS_OPTS="--path.settings ${LS_SETTINGS_DIR} -f /data/logstash/logstash-7.6.0/config/logstash.conf"
 Arguments to pass to java
LS_JAVA_OPTS=""
 pidfiles aren't used the same way for upstart and systemd; this is for sysv users.
LS_PIDFILE=/var/run/logstash.pid
 user and group id to be invoked as
LS_USER=root
LS_GROUP=root
 Enable GC logging by uncommenting the appropriate lines in the GC logging
 section in jvm.options
LS_GC_LOG_FILE=/var/log/logstash/gc.log
 Open file limit
LS_OPEN_FILES=16384
 Nice level
LS_NICE=19
 Change these to have the init script named and described differently
 This is useful when running multiple instances of Logstash on the same
 physical box or vm
SERVICE_NAME="logstash"
SERVICE_DESCRIPTION="logstash"
 If you need to run a command or script before launching Logstash, put it
 between the lines beginning with `read` and `EOM`, and uncomment those lines.
 read -r -d '' PRESTART << EOM
 EOM

5.2 附件logstash.conf

这个文件主要说明的是格式化读取的数据 以及 如何保存到elasticsearch中

 Sample Logstash configuration for creating a simple
 Beats -> Logstash -> Elasticsearch pipeline.
input {
  beats {
    port => 5044
  }
}
filter {
    if [fields][log_type] == "mysql-slow" {
        grok {
            match => ["message", "(?m)^\s+Time:\s+%{TIMESTAMP_ISO8601}\s*\s+User@Host:\s+(?<user>.*)\[%{USERNAME:user}?\]\s*@\s*%{IPORHOST:client}?\s*\[%{IPORHOST:client}?\]\s+Id:\s+%{BASE10NUM}\s*\s+Query_time:\s+%{BASE10NUM:query_time}\s+Lock_time:\s+%{BASE10NUM:lock_time}\s+Rows_sent:\s+%{BASE10NUM:rows_sent}\s+Rows_examined:\s+%{BASE10NUM:rows_examined}\s*(use\s+%{DATA:database};\s*)?SET\s+timestamp=%{BASE10NUM:timestamp};\s*%{GREEDYDATA:sql_stmt}$"]
            keep_empty_captures => true
        }
        date {
            match => ["timestamp", "UNIX"]
            remove_field => ["timestamp"]
        }
        mutate {
            convert => {
                "query_time" => "float"
                "lock_time" => "float"
                "rows_sent" => "integer"
                "rows_examined" => "integer"
            }
            remove_field => ["@version", "beat", "host", "input", "log", "offset", "prospector", "source", "tags"]
        }
    }
    if [fields][log_type] == "mysql-error" {
        grok {
            match => ["message", "(?m)^%{TIMESTAMP_ISO8601:timestamp} %{BASE10NUM} \[%{WORD:error_level}\] %{GREEDYDATA:error_msg}$"]
        }
        date {
            match=> ["timestamp", "ISO8601"]
            remove_field => ["timestamp"]
        }
        mutate {
            remove_field => ["@version", "beat", "host", "input", "log", "offset", "prospector", "source", "tags"]
        }
    }
}
output {
  elasticsearch {
    hosts => ["http://110.110.110.110:10192"]
    index => "%-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    index => "%-qqweixinface-%{+YYYY.MM.dd}"
    user => "qquid_es"
    password => "xiang_ni_123+yidiandian"
  }
}

5.3 附件filebeat.yml

这个文件主要说明了filebeat读取什么log,已经对读取的数据如何处理

 Filebeat Configuration Example 
 This file is an example configuration file highlighting only the most common
 options. The filebeat.reference.yml file from the same directory contains all the
 supported options with more comments. You can use it as a reference.
 You can find the full configuration reference here:
 https://www.elastic.co/guide/en/beats/filebeat/index.html
 For more available modules and options, please see the filebeat.reference.yml sample
 configuration file.
=========================== Filebeat inputs =============================
filebeat.inputs:
 Each - is an input. Most options can be set at the input level, so
 you can use different inputs for various configurations.
 Below are the input specific configurations.
- type: log
   Change to true to enable this input configuration.
  enabled: false
   Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/*.log
    - c:\programdata\elasticsearch\logs\*
  paths:
    - /data/mysql/data/slow.log
  fields:
    log_type: mysql-slow
    db_host: 119.119.119.119
    db_port: 3306
  multiline.pattern: "^ Time:"
  multiline.negate: true
  multiline.match: after
- type: log
  paths:
    - /data/mysql/data/error.log
  fields:
    log_type: mysql-error
    db_host: 119.119.119.119
    db_port: 3306
  multiline.pattern: ^20\d-\d-\dT
  multiline.negate: true
  multiline.match: after
   Exclude lines. A list of regular expressions to match. It drops the lines that are
   matching any regular expression from the list.
  exclude_lines: ['^DBG']
   Include lines. A list of regular expressions to match. It exports the lines that are
   matching any regular expression from the list.
  include_lines: ['^ERR', '^WARN']
   Exclude files. A list of regular expressions to match. Filebeat drops the files that
   are matching any regular expression from the list. By default, no files are dropped.
  exclude_files: ['.gz$']
   Optional additional fields. These fields can be freely picked
   to add additional information to the crawled log files for filtering
  fields:
    level: debug
    review: 1
   Multiline options
   Multiline can be used for log messages spanning multiple lines. This is common
   for Java Stack Traces or C-Line Continuation
   The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  multiline.pattern: ^\[
   Defines if the pattern set under pattern should be negated or not. Default is false.
  multiline.negate: false
   Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
   that was (not) matched before or after or as long as a pattern is not matched based on negate.
   Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
  multiline.match: after
============================= Filebeat modules ===============================
filebeat.config.modules:
   Glob pattern for configuration loading
  path: $/modules.d/*.yml
   Set to true to enable config reloading
  reload.enabled: false
   Period on which files under path should be checked for changes
  reload.period: 10s
==================== Elasticsearch template setting ==========================
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  _source.enabled: false
================================ General =====================================
 The name of the shipper that publishes the network data. It can be used to group
 all the transactions sent by a single shipper in the web interface.
name:
 The tags of the shipper are included in their own field with each
 transaction published.
tags: ["service-X", "web-tier"]
 Optional fields that you can specify to add additional information to the
 output.
fields:
  env: staging
============================== Dashboards =====================================
 These settings control loading the sample dashboards to the Kibana index. Loading
 the dashboards is disabled by default and can be enabled either by setting the
 options here or by using the `setup` command.
setup.dashboards.enabled: false
 The URL from where to download the dashboards archive. By default this URL
 has a value which is computed based on the Beat name and version. For released
 versions, this URL points to the dashboard archive on the artifacts.elastic.co
 website.
setup.dashboards.url:
============================== Kibana =====================================
 Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
 This requires a Kibana endpoint configuration.
setup.kibana:
   Kibana Host
   Scheme and port can be left out and will be set to the default (http and 5601)
   In case you specify and additional path, the scheme is required: http://localhost:5601/path
   IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "localhost:5601"
   Kibana Space ID
   ID of the Kibana Space into which the dashboards should be loaded. By default,
   the Default Space will be used.
  space.id:
============================= Elastic Cloud ==================================
 These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).
 The cloud.id setting overwrites the `output.elasticsearch.hosts` and
 `setup.kibana.host` options.
 You can find the `cloud.id` in the Elastic Cloud web UI.
cloud.id:
 The cloud.auth setting overwrites the `output.elasticsearch.username` and
 `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
cloud.auth:
================================ Outputs =====================================
 Configure what output to use when sending the data collected by the beat.
-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
   Array of hosts to connect to.
  hosts: ["localhost:9200"]
   Optional protocol and basic auth credentials.
  protocol: "https"
  username: "elastic"
  password: "changeme"
----------------------------- Logstash output --------------------------------
output.logstash:
   The Logstash hosts
  hosts: ["localhost:5044"]
   Optional SSL. By default is off.
   List of root certificates for HTTPS server verifications
  ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
   Certificate for SSL client authentication
  ssl.certificate: "/etc/pki/client/cert.pem"
   Client Certificate Key
  ssl.key: "/etc/pki/client/cert.key"
================================ Processors =====================================
 Configure processors to enhance or manipulate events generated by the beat.
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
================================ Logging =====================================
 Sets log level. The default log level is info.
 Available log levels are: error, warning, info, debug
logging.level: debug
 At debug level, you can selectively enable logging only for some components.
 To enable all selectors use ["*"]. Examples of other selectors are "beat",
 "publish", "service".
logging.selectors: ["*"]
============================== X-Pack Monitoring ===============================
 filebeat can export internal metrics to a central Elasticsearch monitoring
 cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
 reporting is disabled by default.
 Set to true to enable the monitoring reporter.
monitoring.enabled: false
 Sets the UUID of the Elasticsearch cluster under which monitoring data for this
 Filebeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
 is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
monitoring.cluster_uuid:
 Uncomment to send the metrics to Elasticsearch. Most settings from the
 Elasticsearch output are accepted here as well.
 Note that the settings should point to your Elasticsearch *monitoring* cluster.
 Any setting that is not set is automatically inherited from the Elasticsearch
 output configuration, so if you have the Elasticsearch output configured such
 that it is pointing to your Elasticsearch monitoring cluster, you can simply
 uncomment the following line.
monitoring.elasticsearch:
================================= Migration ==================================
 This allows to enable 6.7 migration aliases
migration.6_to_7.enabled: true

5.4.附件filebeat.service

这个文件是关于filebeat.service的定义

[Unit]
Description=filebeat.service
[Service]
User=root
ExecStart=/data/filebeat/filebeat-7.4.2-linux-x86_64/filebeat -e -c /data/filebeat/filebeat-7.4.2-linux-x86_64/filebeat.yml
[Install]
WantedBy=multi-user.target

的博客即将同步至腾讯云+社区,邀请大家一同入驻:https://cloud.tencent.com/developer/support-plan?invite_code=3opj47skjx4ws

原文创作:东山絮柳仔

原文链接:https://www.cnblogs.com/xuliuzai/p/14486122.html

文章列表

更多推荐

更多
  • Go安全-二、Go 编程语言 Go 语言规范,The Go playground,Go之旅,关键词,评论,类型,Boolean,数字的,通用编号,具体数字,无符号整数,Signed integers,浮点数,其他数字类型,一串, 在深入研究使用 GO 进行安全
    Golang资深开发

  • Go安全-三、使用文件 File basics,创建空文件,截断文件,获取文件信息,重命名文件,删除文件,打开和关闭文件,检查文件是否存在,检查读写权限,更改权限、所有权和时间戳,硬链接和符号链接,读写,复制文件,Seeking positions in a
    Golang资深开发

  • Go安全-一、Go 安全简介 About Go,Go语言设计,The History of Go,收养与社区,关于Go的常见批评,Go工具链,Go吉祥物,学习Go,为什么要用Go?,为什么使用 Go 进行安全保护?,为什么不使用 Python 呢?,为什么不使用 J
    Golang资深开发

  • Go安全-零、前言 这本书是给谁的,这本书涵盖的内容,充分利用这本书,下载示例代码文件,使用的惯例,联系,评论, 本书涵盖了 Go 编程语言,并解释了如何将其应用于网络安全行业。所涵盖的主题对于红色和蓝色团队、希望编写安全代码的开发人员以及希望保护其
    Golang资深开发

  • Go安全-十、爬虫 爬虫基础,使用 strings 包在 HTTP 响应中查找字符串,使用正则表达式查找页面中的电子邮件地址,从 HTTP 响应中提取 HTTP 头,使用 HTTP 客户端设置 Cookie,在 web 服务器上查找未列出的文件,更改请求的
    Golang资深开发

  • Go安全-六、密码学 散列,散列小文件,散列大文件,安全地存储密码,加密,加密安全伪随机数生成器CSPRNG,Symmetric encryption,AES,非对称加密,生成公钥和私钥对,对邮件进行数字签名,验证签名,TLS,生成自签名证书,创建证书签名请
    Golang资深开发

  • Go安全-七、安全 Shell(SSH) 七、安全 ShellSSH使用 GoSSH 客户端,Authentication methods,使用密码进行身份验证,使用私钥进行身份验证,验证远程主机,通过 SSH 执行命令,启动交互式 shell, 安全外壳(S
    Golang资深开发

  • Go安全-八、暴力破解 Brute forcing HTTP basic authentication,强制使用 HTML 登录表单,强制 SSH,Brute forcing database login, 蛮力攻击,也称为穷举密钥攻击,是指您尝试输入的
    Golang资深开发

  • Go安全-十三、实现漏洞利用 交叉编译,创建绑定壳,创建反向绑定壳,创建 web shell,查找可写文件,更改文件时间戳,更改文件权限,更改文件所有权, 后利用是指渗透测试的一个阶段,其中一台机器已经被利用,代码执行可用。主要任务通常是保持持久性,以便您可以
    Golang资深开发

  • Go安全-十四、总结 重述您所学的主题关于 Go 用法的更多思考,我希望你能从这本书中得到什么,了解法律、道德和技术界限,从这里到哪里去,获得帮助和学习更多,重述您所学的主题到目前为止,在本书中,我们讨论了许多关于Go和信息安全的话题。所涵盖
    Golang资深开发

  • 近期文章

    更多
    文章目录

      推荐作者

      更多