您可以找到著名的好的网络爬虫数据的非常透彻的数据库中robotstxt.org的机器人数据库。不仅仅是匹配利用这个数据将更为有效的机器人的在用户代理。

You can find a very thorough database of data on known "good" web crawlers in the robotstxt.org Robots Database. Utilizing this data would be far more effective than just matching bot in the user-agent.