curl: (56) Recv failure: Connection reset by peer

项目之前跑着一直都很正常,突然获取不到数据了,换了几个不同的网络进行测试,现象为:浏览器可以正常访问获取数据,curl命令行测试报错“curl: (56) Recv failure: Connection reset by peer”,程序无法正常获取数据。

程序后台报错日志

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
2022-05-10 00:16:00.064 [ RenrenScheduler_Worker-4 ] - [ ERROR ] [ io.renren.utils.ScheduleJob : 64 ] - 任务执行失败,任务ID:1487708039686402049
java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at io.renren.utils.ScheduleJob.executeInternal(ScheduleJob.java:54)
at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:75)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: feign.FeignException$InternalServerError: [500 ] during [GET] to [http://renren-biz-server/biz/syn/weather/hour] [BizFeignClient#getWeatherRegionHourList(Map)]: [{"timestamp":1652112960062,"status":500,"error":"Internal Server Error","message":"","path":"/biz/syn/weather/hour"}]
at feign.FeignException.serverErrorStatus(FeignException.java:231)
at feign.FeignException.errorStatus(FeignException.java:180)
at feign.FeignException.errorStatus(FeignException.java:169)
at feign.codec.ErrorDecoder$Default.decode(ErrorDecoder.java:92)
at feign.AsyncResponseHandler.handleResponse(AsyncResponseHandler.java:96)
at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:138)
at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:89)
at com.alibaba.cloud.sentinel.feign.SentinelInvocationHandler.invoke(SentinelInvocationHandler.java:109)
at com.sun.proxy.$Proxy142.getWeatherRegionHourList(Unknown Source)
at io.renren.task.WeatherHourTask.run(WeatherHourTask.java:26)
... 8 common frames omitted
2022-05-10 00:16:00.064 [ RenrenScheduler_Worker-4 ] - [ ERROR ] [ io.renren.utils.Sched

curl 测试报错

1
2
3
4
5
6
--GET
[root@siluxa2 ~]# curl “http://x.x.x.x/china/monitor/city-hour?pollute=AQI&monitorTime=2022-05-09+14”
curl: (56) Recv failure: Connection reset by peer
--POST
[root@siluxa2 ~]# curl -d "pollute=AQI&monitorTime=2022-05-09+14" http://x.x.x.x/china/monitor/city-hour
curl: (56) Recv failure: Connection reset by peer

调整过系统最大链接数,调整过系统最大打开文件数,优化过请求响应时间都未能解决这个问题。

经过多次不同的网络环境,不同的服务器进行测试,可能是因为数据获取太频繁,被网络完全设备判断为爬虫行为进行控制了,尝试添加一个useragent

1
2
3
4
curl -A "User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36" "http://x.x.x.x/china/monitor/city-hour?pollute=AQI&monitorTime=2022-05-09+14"
{"code":0,"msg":"success","data":[{"regionCode":"110000","regionName":"北京市","monitorTime":"2022-05-09 14","lon":"116.346777","lat":"40.064715","primePollute":"—","aqi":33,"pm10":19,"no2":8,"o3":104,"so2":2,"co":0.3,"aqiLevelName":"优","aqiLevel":1,"pm10Level":1,"pm25Level":1,"o3Level":1,"no2Level":1,"so2Level":1,"coLevel":1,"pcpn":0.0,"hum":47,"tmp":17,"tmpLevel":3,"pres":1016,"windDir":"西南风","windDeg":225,"windSpd":18.0,"windLevel":3,"pm2_5":10}
……

至此,总算是找到原因了,在程序中添加一个useragent应该就可以解决掉这个问题了。

ps:顺便推荐几个工具tcping,httping