Yahoo!’s Search Blog announced yesterday that they were making some final changes to their spider, (named “Slurp”), standardizing their crawlers to provide a common DNS signature for identification/authorization purposes.

Previously, Slurp’s requests may have come from IP addresses associated with inktomisearch.com, and now they should all come from IPs associated with domains in this standard syntax:

[something].crawl.yahoo.net

What will this mean to most of us? In most cases, likely nothing. Most sites out there are not likely to be currently performing reverse DNS lookups to check if search engine spiders are actually coming from the IPs/Domains they’re supposed to, except when those spiders get really impolite in requesting too many pages per second. Most people are only identifying bots by their User-Agent strings.

In fact, Yahoo’s provision of this authoritative bot ID syntax is more advanced than Google’s! Google only recommends that people identify their bot (aka “Googlebot”) solely through the User-Agent String — a bit unsatisfactory for a lot of webmasters out there. I’ve heard quite a number of webmasters ask what IP address block to expect the Googlebot requests to originate from, and Google wouldn’t provide them with an authoritative answer.

Of course, one could take a visiting bot’s IP address, say “66.249.65.69″, and perform a Network WHOIS lookup on it to find out if it’s in a block owned by Google. The Network Whois for 66.249.65.69 returns the following info (lookup info provided by Hexillion’s Domain Dossier) :

OrgName: Google Inc.
OrgID: GOGL
Address: 1600 Amphitheatre Parkway
City: Mountain View
StateProv: CA
PostalCode: 94043
Country: US

NetRange: 66.249.64.0 - 66.249.95.255
CIDR: 66.249.64.0/19
NetName: GOOGLE
NetHandle: NET-66-249-64-0-1
Parent: NET-66-0-0-0-0
NetType: Direct Allocation
NameServer: NS1.GOOGLE.COM
NameServer: NS2.GOOGLE.COM
NameServer: NS3.GOOGLE.COM
NameServer: NS4.GOOGLE.COM
Comment:
RegDate: 2004-03-05
Updated: 2007-04-10

OrgTechHandle: ZG39-ARIN
OrgTechName: Google Inc.
OrgTechPhone: +1-650-318-0200
OrgTechEmail: arin-contact

While webmasters could do this lookup for requests for bots displaying the Googlebot user-agent string, it’s still very unsatisfactory because Google does not state that all requests necessarily come from IP blocks that are identifiably owned by Google. So, webmasters would be nervous about blocking something that claimed to be Googlebot yet came from non-Google IP address ranges. After all, it’s possible that Google could have purchased IP addresses and domain names through a proxy in order to perform various types of investigative page requests on sites.

There are cases where hostile dataminers will set their user-agent strings up to masquerade as major search engine spiders, so this newly authoritative method for IDing the bots places Yahoo one step ahead of the game for those webmasters who feel the need to ban the badguys who are scraping their site’s content or requesting pages fast enough to be a defacto denial of service attack.

. . . . . . . . . . . . . . . . . . . .

UPDATE: incrediBILL, one of the moderators at WebmasterWorld, kindly pointed out to me that Matt Cutts had provided the same sort of Googlebot authentication method via the Webmaster Central Blog not long ago. I wish that Google would update their webmaster help section to reflect the same information, if this is indeed intended to be a trustworthy method for authenticating Googlebot. With the instruction only to be found in the blog and not in the actual help section, it still leaves one with the uncomfortable feeling that it’s perhaps an informal method and might still not be depended upon to be true for all cases or it could abruptly change. Hopefully, they’ll update the help pages so everything will be in sync!

This has a couple of interesting ramifications for SEO professionals and those of us who are optimizing our sites for Google, since bot detection systems may now need to be updated and since this may essentially be a new way of committing site/page submissions into Google’s indices. And we all thought automated URL submissions were dead! I’ll explain further….

First of all, quite a few people like to track the movements of bots through their site pages, in order to know the frequency of spider visits, and to confirm that a page has been spidered, period. For sites/pages which have frequent updates happening upon them, it’s also useful to know the date/time the page gets re-spidered and then to see when the updated text will typically appear in the SERPs. Also, some folks have set their robots.txt to disallow spiders into sections of their sites for various reasons.

So, this change in Google’s spidering functionality will be important for you if you have AdSense ads running on your site. You’ll want to update your robots.txt file to reflect the AdSense bot’s user agent string, and you’ll also want to make sure this user-agent string definition is present in the logfile analysis systems you’re using to track spider activity on your site.

Many of us are using the Web Robots Database at Robotstxt.org in order to identify bots and spiders passing through our sites, and it’s a great resource for all information about the robots exclusion protocol and related matters — Google even cites them as an information resource throughout their webmaster info pages. However, the Robots Database has not been updated to include the definition of the AdSense bot as of the time that I’m writing this. (I’ve just reported the bot identification information over to them to add in, so hopefully this won’t be the case for long.)

(Some webmasters and systems are using the IP address of the bots instead of the User-Agent Strings, but I consider the preferred method to be to use the User Agent for this purpose. Otherwise, you risk counting a search engine employee who is browsing your site during their coffee break to be their spider visiting you!)

Matt Cutts is apparently informally referring to this bot as “Mediabot” or “Media Bot”, but the bot is currently declaring itself with this User-Agent String:

Mediapartners-Google/2.1

If you want to specifically disallow this bot from some section of your site, you should wildcard the bot version number at the end in your robots.txt file like this:

User-agent: Mediapartners-Google*
Disallow: /dont-crawl-this-uri-on-my-site

Another interesting point is raised due to the indexing of the Mediabot-spidered content: doesn’t this basically provide a new way to, errrr, automatically submit pages to Google?!?

If you manually submit your site to Google using their submission form, you’re only allowed to provide the top-level domain name of your site. Using that method, Googlebot will initially visit your homepage, and likely only crawl through one or two levels of links out from the homepage in that initial spidering visit. If you’ve got a really deep site with thousands of pages of content, Googlebot might later revisit the site to try to spider more deeply, in a widening circle out from the homepage. This existing process could utlimately take quite some time before all of your content gets spidered and can begin appearing in the SERPs.

I’m thinking that for cases like that where you have a lot of pages on a new or non-indexed site, adding the Google ads onto all your pages might actually result in them getting initially spidered more rapidly.

Also, quite a lot more pages could potentially get indexed if they have the ads on them, since there are situations where Googlebot will not or cannot spider pages on sites. For instance, if your site content is accessible primarily only through a submission form on your homepage, or through Java/Javascripted menus, or you have only some Flash-enabled navigation system (of course, no SEO professional worth his or her salt would use a Flash-only nav system!) — if your site pages aren’t navigable through regular links displayed on your pages, Googlebot would otherwise never find and index their content.

But, now pages that were only accessible to users through a submission form on your site could potentially now get indexed and appear in SERPs if they have AdSense ads on them!

This is mostly a good thing for those of us working hard to expose content through the SEs, but I bet it could create some havoc for webmasters who are AdSense publishers and who are taken unaware by the potential sudden influx in traffic which pounds their databases as pages suddenly become visible in SERPs. Fun problem to have, though!

Google apparently uses other bots specifically for harvesting other types of content as well. Two that I’ve come across include a bot which grabs images from websites to use in their Google Images section, and a bot for gathering RSS feeds for use in personalized Google homepage, or in the Google Reader.

These are also not identified in Robotstxt.org yet, but their User-Agent strings are as follows:

User-agent: Googlebot-ImageUser-agent: Feedfetcher-Google

Note: Feedfetcher ignores robots.txt exclusion files! Google does this because:

Feedfetcher retrieves feeds only after users have explicitly added them to their Google homepage or Google Reader. Feedfetcher behaves as a direct agent of the human user, not as a robot, so it ignores robots.txt entries. Feedfetcher does have one special advantage, though: because it’s acting as the agent of multiple users, it conserves bandwidth by making requests for common feeds only once for all users.

Do you know if Google is using other specialized bots for other sections of their site or other types of media? If so, I’d be interested in hearing about it.