Yahoo’s Recent Spider Improvement Beats Google’s
Yahoo!’s Search Blog announced yesterday that they were making some final changes to their spider, (named “Slurp”), standardizing their crawlers to provide a common DNS signature for identification/authorization purposes.
Previously, Slurp’s requests may have come from IP addresses associated with inktomisearch.com, and now they should all come from IPs associated with domains in this standard syntax:
[something].crawl.yahoo.net
Possible Related Posts
Posted by Chris of Silvery on 06/06/2007
Permalink | | 
Print
| Trackback | Comments Off on Yahoo’s Recent Spider Improvement Beats Google’s | Comments RSS
Filed under: Google, Spiders, Yahoo bot-detection, bots, Googlebot, slurp, spidering, Spiders, user-agents
