北京第三期_质量监控_20181223

[seveniruby (思寒)]

质量监控是什么

质量数据的采集

• app埋点 插桩
• app的代理
• app的数据转发 借助于多套环境
• 服务端log

演练环境

• 待测系统：Jenkins http://jenkins.testing-studio.com:8080/
• 数据记录：使用反向代理或者正向代理来完成数据采集
• 反向代理：可追踪用户身份 http://jenkins.testing-studio.com:9001/
• 公共ELK环境：http://elk.testing-studio.com:5601

数据结构

不埋点的情况下能抓到的数据

• 接口请求：url、status
• 用户身份： ip、cookie
• 行为历史：referer
• 版本号：ua

雪球app数据

GET /v5/stock/realtime/quotec.json?_t=1GENYMOTION60139ebef43ea8c54751bea8eea5494c.3708411383.1545532684060.1545532698627&_s=978a0e&is_dalay_hk=false&symbol=SH000001%2CSZ399001%2CSZ399006 HTTP/1.1
Cookie\txq_a_token=06574dd455c8f66976405827089d5d845f537d57;u=3708411383
User-Agent\tXueqiu Android 11.10.2
Accept-Encoding\tgzip
Accept-Language\ten-US,en;q=0.8,zh-CN;q=0.6,zh;q=0.4
Host\tstock.xueqiu.com
Connection\tKeep-Alive

web浏览数据

GET /job/AllureDemo/18/ HTTP/1.1
Host: jenkins.testing-studio.com:8080
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Referer: http://jenkins.testing-studio.com:8080/job/AllureDemo/
Accept-Encoding: gzip, deflate
Accept-Language: zh-CN,zh;q=0.9,en-US;q=0.8,en;q=0.7
Cookie: jenkins-timestamper-offset=-28800000; _ga=GA1.2.2053923289.1527161186; JSESSIONID.d1bd489a=node01cpya6qlzrvpxxg6x2e79tn16433.node0; JSESSIONID.dd4a903c=node0tr2b2ezvj20z4sj43ugeb4yo131.node0; screenResolution=1280x800
Connection: close

反向代理nginx

events { worker_connections 1024; }
http {
 log_format access_log_json '{"log_time": "$time_iso8601", "uid": "$cookie_uid", "req": "$request_uri", "code": "$status"}'; 
   access_log /usr/share/nginx/html/access_log_json.log access_log_json;
userid         on;
userid_name    uid;
server {
    listen       80;
    location / {
        proxy_pass    http://jenkins.testing-studio.com:8080/;
        proxy_set_header Host $host;
        proxy_redirect off;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_connect_timeout 60;
        proxy_read_timeout 600;
        proxy_send_timeout 600;
    }
}
}

使用docker启动

docker run \\
-d \\
--name nginx \\
-v $PWD/www:/usr/share/nginx/html \\
-v $PWD:/etc/nginx/ \\
-p 9001:80 \\
registry.docker-cn.com/library/nginx

原来的网站

GET /job/demo/ HTTP/1.1
Host: jenkins.testing-studio.com:8080
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Referer: http://jenkins.testing-studio.com:8080/
Accept-Encoding: gzip, deflate
Accept-Language: zh-CN,zh;q=0.9,en-US;q=0.8,en;q=0.7
Cookie: JSESSIONID.dd4a903c=node03rgjv88up15dvc344ntauzmk136.node0; screenResolution=1280x800
Connection: close

反向代理

GET /job/demo/6/ HTTP/1.1
Host: jenkins.testing-studio.com:9001
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Referer: http://jenkins.testing-studio.com:9001/job/demo/
Accept-Encoding: gzip, deflate
Accept-Language: zh-CN,zh;q=0.9,en-US;q=0.8,en;q=0.7
Cookie: JSESSIONID.dd4a903c=node03rgjv88up15dvc344ntauzmk136.node0; screenResolution=1280x800; uid=rBIABVwe/WnB3AAGAwMEAg==
Connection: close

ELK

• logstash
• ElasticSearch
• kibana

搭建ELK

docker run \\
--name elk \\
-d \\
-e LOGSTASH_START=0 \\
-p 5601:5601 \\
-p 9200:9200 \\
-p 5044:5044 -d \\
registry.docker-cn.com/sebp/elk

Logstash

docker run -it --rm registry.docker-cn.com/library/logstash -e 'input { stdin { } } output { stdout { } }'
docker run -it --rm registry.docker-cn.com/library/logstash -e 'input { stdin { } } output { elasticsearch {
hosts => ["jenkins.testing-studio.com:9200"]
index => "logstash-demo-%{+YYYY.MM.dd}"
} }'

ES

http://jenkins.testing-studio.com:9200/_cat/health?v

epoch      timestamp cluster       status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1545542406 05:20:06  elasticsearch yellow          1         1      6   6    0    0        5             0                  -                 54.5%

epoch      timestamp cluster       status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1545542780 05:26:20  elasticsearch green           1         1      6   6    0    0        0             0                  -                100.0%

状态为yellow的可能原因

• 磁盘不够
• 节点不够

修改number_of_replicas为0

  "index.number_of_replicas": "0"

csv数据

csv.conf

input {
    file {
        path => "/data/*.csv"
        start_position => beginning
    }
}
filter {
   csv{
        columns =>[ "log_time", "user", "api", "status", "version"]
   }
  date {
        match => ["log_time", "yyyy-MM-dd HH:mm:ss"]
        timezone => "Asia/Shanghai"
    }
}
output {
    elasticsearch {
        hosts => ["elk.testing-studio.com:9200"]
        index => "logstash-csv-%{+YYYY.MM.dd}"
    }
}

在当前目录下执行

docker run -it --rm \\
-v $PWD/data/:/data/  \\
-v $PWD:/conf/  \\
registry.docker-cn.com/library/logstash -f /conf/csv.conf --verbose

构造数据

while true
do
version=$([ $((RANDOM%5)) -ge 1 ] && echo debug || echo test)
version=${version}_3.$((RANDOM%3))
api=api/$((RANDOM%5)).json
status=$((RANDOM%5))00
ip=192.168.0.1$((RANDOM%5))$((RANDOM%5))
echo $(date +"%Y-%m-%d %H:%M:%S"),${ip},${api},${status},${version} | tee -a $(
date +%Y%m%d%H%M).csv
sleep 0.$((RANDOM%5))
done

JSON数据

json.conf

input {
    file {
        path => "/data/*.log"
        codec => "json"
    }
}
filter {
  date {
        match => ["log_time", "ISO8601"]
        timezone => "Asia/Shanghai"
    }
}
output {
    elasticsearch {
        hosts => ["elk.testing-studio.com:9200"]
        index => "logstash-json-%{+YYYY.MM.dd}"
    }
}

执行数据导入命令

docker run -it --rm \\
-v $PWD/../nginx/www/:/data/  \\
-v $PWD:/conf/  \\
registry.docker-cn.com/library/logstash -f /conf/json.conf --verbose

基础数据分析

• 数据分类
• 检索
• 绘图
• 面板

饼图

测试覆盖

• /login?user=xx&password=xx&type=mobile
• /login?user=xx&password=xx&type=mail
• /search?type=user
• /search?type=stock&q=sogo
• /search?type=stock&q=alibaba
• /search?type=all

变化为

• /login/type=mobile
• /login/type=mail
• /search?type=user
• /search?type=stock
• /search?type=all
• /submit/post

场景

A用户在购买的时候，出现购买失败，ui有反馈toast，接口层也有反馈errorcode
nginx log可以拦截 get post

• springboot基于jvm，jvm有javaagent等类似的插桩机制
  研发硬编码
• 字节码插桩 if else jacoco

用户 → app → jacoco → http → nginx → springboot → jacoco → logstash → elk → kibana

• 插桩技术
• jacoco数据转成调用链
• 功能与代码调用链进行关联
• 代码变更反推功能回归和用例覆盖

练习1

根据csv的索引，制作一个x轴是时间，y轴是不同状态码的趋势图，把截图贴到回复

练习2

根据json索引，制作一个内测是状态码，中层是req，要包含 /name，外层是用户,

课后作业

• 从零开始构造自己的数据，可以是任意格式，比如csv、json、kafka、http请求等，然后把数据存入到ELK公共演练环境，名字为logstash-你的名字-日期
• 制作分析报表，完成线图、趋势图、多维度柱形图的分析，图形名字保存为 你的名字_作用