线下班第一期 ELK 质量监控课程

ELK安装

docker run \\
--name elk -d \\
-e ES_HEAP_SIZE="1g" -e MAX_MAP_COUNT=262144 \\
-p 5601:5601 -p 9200:9200 -p 5044:5044 \\
registry.docker-cn.com/sebp/elk

LogStash

https://store.docker.com/images/logstash

docker pull logstash
#中国区加速
docker pull registry.docker-cn.com/library/logstash

logstash的配置

input {
    file {
        path => "/data/*.csv"
        start_position => beginning
     }
}
filter {
   csv{
        columns =>[ "log_time", "user", "api", "status", "version"]
   }
  date {
        match => ["log_time", "yyyy-MM-dd HH:mm:ss"]
        timezone => "Asia/Shanghai"
    }
}
output {
    elasticsearch {
        hosts => ["192.168.31.99:9200"]
        index => "logstash-rc-%{+YYYY.MM.dd}"
    }
}

启动logstash

docker run -it --rm \\
-v /Users/seveniruby/temp/logstash/logstash-2.3.2/conf/:/config-dir  \\
-v /Users/seveniruby/temp/logstash/data/:/data \\
logstash -f /config-dir/csv.conf

检查索引

http://127.0.0.1:9200/_cat/indices?v

创建测试数据

while true
do
  version=$([ $((RANDOM%5)) -ge 1 ] && echo debug || echo test)
  version=${version}_3.$((RANDOM%3))
  api=api/$((RANDOM%5)).json
  status=$((RANDOM%5))00
  ip=192.168.0.1$((RANDOM%5))$((RANDOM%5))
  echo $(date +"%Y-%m-%d %H:%M:%S"),${ip},${api},${status},${version} | tee -a  $(date +%Y%m%d%H%M).csv
  sleep 0.$((RANDOM%5))
done

Kibana使用

user:"192.168.0.110" AND NOT status:200

常见问题

On Linux, use sysctl vm.max_map_count on the host to view the current value, and see Elasticsearch’s documentation on virtual memory for guidance on how to change this value. Note that the limits must be changed on the host; they cannot be changed from within a container.

If using Docker for Mac, then you will need to start the container with the MAX_MAP_COUNT environment variable (see Overriding start-up variables) set to at least 262144 (using e.g. docker’s -e option) to make Elasticsearch set the limits on mmap counts at start-up tim

关闭