Yahoo’s Recent Spider Improvement Beats Google’s
Yahoo!’s Search Blog announced yesterday that they were making some final changes to their spider, (named “Slurp”), standardizing their crawlers to provide a common DNS signature for identification/authorization purposes.
Previously, Slurp’s requests may have come from IP addresses associated with inktomisearch.com, and now they should all come from IPs associated with domains in this standard syntax:
[something].crawl.yahoo.net
Popularity: 55% [?]
Possible Related Posts
Posted by Chris Silver Smith of Netconcepts on 06/06/2007 | Permalink | 
Email
| 
Print
Trackback | Comments (0) | Comments RSS |
Filed under: Google, Spiders, Yahoobot-detection, bots, Googlebot, slurp, spidering, Spiders, user-agents
