且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

如何提高Scrapy的爬行速度?

更新时间:2023-10-20 11:41:58

尝试调整 CONCURRENT_ITEMSCONCURRENT_REQUESTSCONCURRENT_REQUESTS_PER_DOMAIN 和其他设置.

Try tuning the CONCURRENT_ITEMS, CONCURRENT_REQUESTS, CONCURRENT_REQUESTS_PER_DOMAIN and other settings.

有关设置的完整列表,请参阅 http://doc.scrapy.org/en/latest/topics/settings.html

For full list of settings, see http://doc.scrapy.org/en/latest/topics/settings.html