南山草舍

0%

curl: (56) Recv failure: Connection reset by peer

发表于 更新于 分类于

项目之前跑着一直都很正常，突然获取不到数据了，换了几个不同的网络进行测试，现象为：浏览器可以正常访问获取数据，curl命令行测试报错“curl: (56) Recv failure: Connection reset by peer”，程序无法正常获取数据。

程序后台报错日志

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
2022-05-10 00:16:00.064 [ RenrenScheduler_Worker-4 ] - [ ERROR ] [ io.renren.utils.ScheduleJob : 64 ] - 任务执行失败，任务ID：1487708039686402049
java.lang.reflect.InvocationTargetException: null
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at io.renren.utils.ScheduleJob.executeInternal(ScheduleJob.java:54)
        at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:75)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: feign.FeignException$InternalServerError: [500 ] during [GET] to [http://renren-biz-server/biz/syn/weather/hour] [BizFeignClient#getWeatherRegionHourList(Map)]: [{"timestamp":1652112960062,"status":500,"error":"Internal Server Error","message":"","path":"/biz/syn/weather/hour"}]
        at feign.FeignException.serverErrorStatus(FeignException.java:231)
        at feign.FeignException.errorStatus(FeignException.java:180)
        at feign.FeignException.errorStatus(FeignException.java:169)
        at feign.codec.ErrorDecoder$Default.decode(ErrorDecoder.java:92)
        at feign.AsyncResponseHandler.handleResponse(AsyncResponseHandler.java:96)
        at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:138)
        at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:89)
        at com.alibaba.cloud.sentinel.feign.SentinelInvocationHandler.invoke(SentinelInvocationHandler.java:109)
        at com.sun.proxy.$Proxy142.getWeatherRegionHourList(Unknown Source)
        at io.renren.task.WeatherHourTask.run(WeatherHourTask.java:26)
        ... 8 common frames omitted
2022-05-10 00:16:00.064 [ RenrenScheduler_Worker-4 ] - [ ERROR ] [ io.renren.utils.Sched

curl 测试报错

1
2
3
4
5
6
--GET
[root@siluxa2 ~]# curl “http://x.x.x.x/china/monitor/city-hour?pollute=AQI&monitorTime=2022-05-09+14”
curl: (56) Recv failure: Connection reset by peer
--POST
[root@siluxa2 ~]# curl -d "pollute=AQI&monitorTime=2022-05-09+14" http://x.x.x.x/china/monitor/city-hour
curl: (56) Recv failure: Connection reset by peer

调整过系统最大链接数，调整过系统最大打开文件数，优化过请求响应时间都未能解决这个问题。

经过多次不同的网络环境，不同的服务器进行测试，可能是因为数据获取太频繁，被网络完全设备判断为爬虫行为进行控制了，尝试添加一个useragent

1
2
3
4
curl  -A "User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36" "http://x.x.x.x/china/monitor/city-hour?pollute=AQI&monitorTime=2022-05-09+14"

{"code":0,"msg":"success","data":[{"regionCode":"110000","regionName":"北京市","monitorTime":"2022-05-09 14","lon":"116.346777","lat":"40.064715","primePollute":"—","aqi":33,"pm10":19,"no2":8,"o3":104,"so2":2,"co":0.3,"aqiLevelName":"优","aqiLevel":1,"pm10Level":1,"pm25Level":1,"o3Level":1,"no2Level":1,"so2Level":1,"coLevel":1,"pcpn":0.0,"hum":47,"tmp":17,"tmpLevel":3,"pres":1016,"windDir":"西南风","windDeg":225,"windSpd":18.0,"windLevel":3,"pm2_5":10}
……

至此，总算是找到原因了，在程序中添加一个useragent应该就可以解决掉这个问题了。

ps：顺便推荐几个工具tcping，pathping，httping，psping