charles抓取的内容与python+requests获取的内容不一样(python+requests获取的内容是不完整的)

charles抓取的内容与python+requests获取的内容不一样(python+requests获取的内容是不完整的)
1、python脚本内容如下

import requests
import json

# 请求地址
url = "https://xiapi.xiapibuy.com//api/v2/search_items/?by=relevancy&categoryids=8436&keyword=%E5%AF%B5%E7%89%A9%E8%A1%A3%E6%9C%8D&limit=50&newest=0&order=desc&page_type=search&search_prefill=533&skip_autocorrect=1&version=2"
headers = {

    "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36",
    "content-type": "application/json; charset=utf-8",
    "accept-encoding": "gzip, deflate, br",
    "accept-language": "zh-CN,zh;q=0.9",
    "x-shopee-language": "zh-Hant",
    "accept": "*/*",
    "sec-fetch-mode": "cors",
    'Connection' : 'Keep-Alive',
    "sec-fetch-dest": "empty",
    "sec-fetch-site": "same-origin",
    "cookie": '_gcl_au=1.1.1950202641.1614253363; __BWfp=c1614255385494x4861ad98c; SPC_IA=-1; SPC_EC=-; SPC_U=-; REC_T_ID=24860de2-7763-11eb-bd88-4cd98f6396a5; SPC_F=kIgpBjzooYwbSeny10VD6OAmFIgeOOg3; REC_T_ID=2493be6c-7763-11eb-b4a3-f8f21e1af482; SPC_SI=mall.17RfWei0O6TM7UTuozlgZRk0BW6lLDn4; csrftoken=h5XPkI4xC55yw94TnEN874g3QAAJN6KV; welcomePkgShown=true; SPC_CT_7f3c470f="1614504896.PiZPn+qb4Gb/c3/vSaDeWb+1Kd7KhVAU7zBIHfZbosw="; SPC_CT_cf0618be="1614504896.QasEq9Z0nUzm/kg3plAS8NfpKq6gH2NdCcQ5zRHz+qs="; SPC_CT_5b89d0d0="1614504896.tdhrk1hM1L7/TfwNNkrJFGRMgEjx0/OW5jZt6cNcNXo="; SPC_CT_9507a35d="1614504896.AwSBD6+9WJ5/RAU52y5Lb0Di/HeHFtWcGdoKLofxm5Q="; SPC_CT_e7a3420c="1614504913.4hgQsHsbTrPSr0UTX11hJjYWbq6FmLv/hHbnCXpfSSs="; SPC_CT_f4802b3d="1614504989.vmuSBai4lwfpDp6llJszfGj5gvU5KxH9bQvZmRTJA1o="; SPC_CT_721da2ed="1614504995.fBXlX3MPuvpHDaI4FicMV8KkixqVs4Wm0++Bm751AnA="; SPC_CT_99fc082e="1614505158.SAPj9KV+7VhSJKLpH179u0ITsVlNK8AZNSItAHq/+9g="; SPC_CT_ef220289="1614505191.g3pkOsbL3/6HFr6TOrdMu2PxP2FtcVmioCgImACSvoA="; _med=refer; SPC_CT_f7b18e6e="1614507953.i7wqeaW457WGDw7BnAhwvVAFcjJH8eTBPagiRpawONk="; SPC_R_T_ID="HNFzW4Tiu1UQBC6RcW2I+etJblCalmHZWW6XJ7TkTkMriKgGP1+n6NthWAI44MZ+YV2zy4gvhS7XquqyrsWUczB3X0zp8Upo4Jfayb+8Wsg="; SPC_T_IV="/g4WEECC+1erG7aePadPTQ=="; SPC_R_T_IV="/g4WEECC+1erG7aePadPTQ=="; SPC_T_ID="HNFzW4Tiu1UQBC6RcW2I+etJblCalmHZWW6XJ7TkTkMriKgGP1+n6NthWAI44MZ+YV2zy4gvhS7XquqyrsWUczB3X0zp8Upo4Jfayb+8Wsg="'
}

response = requests.get(url, headers)

res = response.text
print(res)

获取到的数据:

charles抓取到的数据如下:

  • 建议让requests发送的请求也通过charles抓取一下,跟页面访问抓取的请求对比看看差别在哪。
  • requests本身是应该是不会对返回的数据做这种截取操作的,猜测可能是请求中的个别参数不同导致接口对数据做了处理

requests发送的请求通过charles抓取的数据也是一样的,在pycharm和charles看到的数据是不完整的



image

把经过charles的两个请求转成curl命令对比下,别用人眼去看,用diff对比工具。大部分都是因为某个header不同

我的意思不是要通过charles来对比结果,而是要对比发送出去的请求数据
两者请求数据细节上的区别可能会造成系统对返回结果处理的不同,进而导致字段内容的不同