Docker 部署ELK6.6

# 一、下载镜像

注：660为elk版本,这里是直接集成elk三个服务至一个容器中

这里我们使用elk集成镜像，地址：https://hub.docker.com/r/sebp/elk/tags

[root@centos-mq ~]# docker pull sebp/elk:660

# 二、启动

[root@centos-mq ~]# echo "vm.max_map_count=262144" > /etc/sysctl.conf[root@centos-mq ~]# sysctl -p[root@centos-mq ~]# docker run -dit --name elk \
    -p 5601:5601 \
    -p 9200:9200 \
    -p 5044:5044 \
    -v /opt/elk-data:/var/lib/elasticsearch \
    -v /etc/localtime:/etc/localtime \
    sebp/elk:660

1
2
3
4
5
6
7

说明：-p 指定映射端口，5601kibana访问，9200es端口，5044 logstash收集日志端口；-v 指定es数据目录

# 三、访问

启动后等待数据初始化后，浏览器输入：http://10.10.0.13:5601,可看到kibana web界面

# 四、文件目录

通过docker exec -it elk /bin/bash可进入容器中，具体各服务配置文件路径如下

 [root@centos-mq ~]# docker exec -it elk /bin/bash
  /etc/logstash/        ## logstash 配置文件路径
  /etc/elasticsearch/   ##es 配置文件路径
  /var/log/             ## 日志路径

1
2
3
4

# 五、通过filebeat收集java

filebeat部署，版本最好与elk一直，这里也选择6.6.0版本，filebeat部署在应用所在服务器，进行日志收集，日志样例；

2019-06-19 14:34:23.261 [http-nio-8090-exec-7] INFO  com.one.api.user.controller.UserBacklogController - [李强]:获取待办数
2019-06-16 09:36:54.083 [http-nio-8090-exec-6] ERROR com.one.common.exception.OneGlobalExceptionHandler - 【000000系统异常】: URL : http://172.16.223.53:8090/api/backlog/handle ERROR : {}
java.lang.NullPointerException: null
        at com.one.api.contract.service.ContractProcessService.getManagementFieldDataDto(ContractProcessService.java:2394)
        at com.one.api.contract.service.ContractProcessService.getContractFieldData(ContractProcessService.java:2190)
        at com.one.api.contract.service.ContractProcessService.leasePass(ContractProcessService.java:1051)
        at com.one.api.contract.service.ContractProcessService.approvalPass(ContractProcessService.java:530)
        at com.one.api.contract.service.ContractProcessService$$FastClassBySpringCGLIB$$def3f504.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
        at com.one.api.contract.service.ContractProcessService$$EnhancerBySpringCGLIB$$e6bec33f.approvalPass(<generated>)
        at com.one.api.process.flowable.listener.EndTaskHandler.notify(EndTaskHandler.java:77)
        at com.one.api.process.flowable.listener.EndTaskHandler$$FastClassBySpringCGLIB$$cc1052a6.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
        at com.one.api.process.flowable.listener.EndTaskHandler$$EnhancerBySpringCGLIB$$a8a27165.notify(<generated>)
        at org.flowable.engine.impl.delegate.invocation.ExecutionListenerInvocation.invoke(ExecutionListenerInvocation.java:35)
        at org.flowable.engine.impl.delegate.invocation.DelegateInvocation.proceed(DelegateInvocation.java:35)
        at org.flowable.engine.impl.delegate.invocation.DefaultDelegateInterceptor.handleInvocation(DefaultDelegateInterceptor.java:26)
        at org.flowable.engine.impl.bpmn.listener.DelegateExpressionExecutionListener.notify(DelegateExpressionExecutionListener.java:45)
        at org.flowable.engine.impl.bpmn.listener.ListenerNotificationHelper.executeExecutionListeners(ListenerNotificationHelper.java:79)
        at org.flowable.engine.impl.agenda.AbstractOperation.executeExecutionListeners(AbstractOperation.java:78)
        at org.flowable.engine.impl.agenda.AbstractOperation.executeExecutionListeners(AbstractOperation.java:69)
        at org.flowable.engine.impl.agenda.ContinueProcessOperation.executeSynchronous(ContinueProcessOperation.java:141)
        at org.flowable.engine.impl.agenda.ContinueProcessOperation.continueThroughFlowNode(ContinueProcessOperation.java:113)
        at org.flowable.engine.impl.agenda.ContinueProcessOperation.continueThroughSequenceFlow(ContinueProcessOperation.java:311)
        at org.flowable.engine.impl.agenda.ContinueProcessOperation.run(ContinueProcessOperation.java:79)
        at org.flowable.engine.impl.interceptor.CommandInvoker.executeOperation(CommandInvoker.java:88)
        at org.flowable.engine.impl.interceptor.CommandInvoker.executeOperations(CommandInvoker.java:72)
        at org.flowable.engine.impl.interceptor.CommandInvoker.execute(CommandInvoker.java:56)
        at org.flowable.engine.impl.interceptor.BpmnOverrideContextInterceptor.execute(BpmnOverrideContextInterceptor.java:25)
        at org.flowable.common.engine.impl.interceptor.TransactionContextInterceptor.execute(TransactionContextInterceptor.java:53)
        at org.flowable.common.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:71)
        at org.flowable.common.spring.SpringTransactionInterceptor$1.doInTransaction(SpringTransactionInterceptor.java:49)
        at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
        at org.flowable.common.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:46)
        at org.flowable.common.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:30)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

# 5.1 下载安装

[root@centos-mq ~]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.6.0-x86_64.rpm
[root@centos-mq ~]# rpm -ivh filebeat-6.6.0-x86_64.rpm

1
2

# 5.2 配置收集java日志

[root@vanje-dev02 ~]# vim /etc/filebeat/filebeat.yml 
#=========================== Filebeat inputs ============================
filebeat.inputs:- type: log
  enabled: true
  paths:
    - /apps/oneJars/himalaya/logs/one.log     ## 日志路径
  tags: ["one-himalaya"]　　　　　　　　　　　　　## 标签，用于判断
  multiline.pattern: '^\d{4}-\d{2}-\d{2}'　　 ## 匹配日志开头
  multiline.negate: true　　　　　　　　　　　　 ## 日志合并
  multiline.match: after
  
  
#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
#hosts: ["localhost:9200"]    ## 注释 这里是配置采集的日志存放的方式，我们先经过logstash处理，所以这里注释

#----------------------------- Logstash output --------------------------------
output.logstash:
 hosts: ["10.10.0.13:5044"]  ## 采集日志输出到logstash，ip为logstash服务ip

## 启动filebeat
[root@vanje-dev02 ~]# systemctl restart filebeat

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

# 5.3 logstash配置

以下配置只是收集"2019-06-19 14:34:23.261 [http-nio-8090-exec-7] INFO com.one.api.user.controller.UserBacklogController - [李强]:获取待办数"格式日志，用来分析用户使用时间及姓名

## 定过滤指定日志，没用的日志我们这里不收集，正常是要收集所有，vim /etc/logstash/conf.d/02-beats-input.conf
input {
  beats {
    port => 5044
  }
}
filter {
  #if "one-himalaya" in [tags] {
  if [message] =~ '获取待办数' {
    grok {
      patterns_dir => ["/etc/logstash/patterns"]
      match => {
        "message" => "%{DAYTIME:day}\s*%{THREAD:thread}\s%{LOGEVL:level}\s*%{JAVACLASS:class}\s*- \[%{JAVALOGMESSAGE:logmessage}\]:"
         }
    }
  }
}
output {
 # if "one-himalaya" in [tags] {
  if [message] =~ '获取待办数' {
    elasticsearch {
      hosts => ["172.16.223.55:9200"]
      index => "one-himalaya10-15-%{+YYYY.MM.dd}"
    }
  }
}## 自定义匹配规则vim /etc/logstash/patternsDAYTIME \d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{1,2}.\d{1,4}LOGEVL [a-zA-Z]{4}NUM \d{1}THREAD \[http-nio-\d{4}-exec-\d{1,3}\]## 重启logstash/etc/init.d/logstash restart

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

提示

说明：实际生产中，我们会对收集的日志做各种分析提取，需要灵活使用grok 正则匹配来提取自己想要的数据。可以通过Kibana Dev Tools>>Grok Debugger工具进行调式，例如：

# 六、kibana查看日志

打开kibana web界面，跟据提示创建索引，即可看到已经收集的日志信息：day logmessage 这两个字段，为切割日志自定义的字段，这些字段跟据自己所需灵活定义，以区分切割的数据。

编辑

上次更新: 2023/03/13, 16:47:57

← Docker 部署Gitlab11.10.4 Docker 部署ELK之Sentinl日志报警→