ELK安装
docker run \\
--name elk -d \\
-e ES_HEAP_SIZE="1g" -e MAX_MAP_COUNT=262144 \\
-p 5601:5601 -p 9200:9200 -p 5044:5044 \\
registry.docker-cn.com/sebp/elk
LogStash
https://store.docker.com/images/logstash
docker pull logstash
#中国区加速
docker pull registry.docker-cn.com/library/logstash
logstash的配置
input {
file {
path => "/data/*.csv"
start_position => beginning
}
}
filter {
csv{
columns =>[ "log_time", "user", "api", "status", "version"]
}
date {
match => ["log_time", "yyyy-MM-dd HH:mm:ss"]
timezone => "Asia/Shanghai"
}
}
output {
elasticsearch {
hosts => ["192.168.31.99:9200"]
index => "logstash-rc-%{+YYYY.MM.dd}"
}
}
启动logstash
docker run -it --rm \\
-v /Users/seveniruby/temp/logstash/logstash-2.3.2/conf/:/config-dir \\
-v /Users/seveniruby/temp/logstash/data/:/data \\
logstash -f /config-dir/csv.conf
检查索引
http://127.0.0.1:9200/_cat/indices?v
创建测试数据
while true
do
version=$([ $((RANDOM%5)) -ge 1 ] && echo debug || echo test)
version=${version}_3.$((RANDOM%3))
api=api/$((RANDOM%5)).json
status=$((RANDOM%5))00
ip=192.168.0.1$((RANDOM%5))$((RANDOM%5))
echo $(date +"%Y-%m-%d %H:%M:%S"),${ip},${api},${status},${version} | tee -a $(date +%Y%m%d%H%M).csv
sleep 0.$((RANDOM%5))
done
Kibana使用
user:"192.168.0.110" AND NOT status:200
常见问题
On Linux, use sysctl vm.max_map_count on the host to view the current value, and see Elasticsearch’s documentation on virtual memory for guidance on how to change this value. Note that the limits must be changed on the host; they cannot be changed from within a container.
If using Docker for Mac, then you will need to start the container with the MAX_MAP_COUNT environment variable (see Overriding start-up variables) set to at least 262144 (using e.g. docker’s -e option) to make Elasticsearch set the limits on mmap counts at start-up tim