It is a good idea to have some knowledge of the factors that influence the crawling of a site and its successful indexing before the site ranks on the SERPs. The site can be built in a user friendly way that allows the spiders to know what to crawl and how frequently to crawl.

Crawling Factors

1. Links:
   All major search engines crawl the web through link structures. If a site has a good link structure starting broadly from the top and going down into the category and sub category level, with all the money pages three to four clicks away from the home page, the bots would find crawling the site a lot easy. Placing a sitemap on the home page further assists the bots to find all the content on the site.
2. Content Freshness and Updates:
   This is one of the best ways to keep the bots coming back to your site regularly. It is vital to have fresh content updated regularly on a site. A blog will go a long way in achieving this. To a googlebot, new content is a sign of attaching more importance to the site by visiting it more often.

   There is a Query Deserves Freshness (QDF) component in Google’s algorithm that awards sites with updated content (news sites for example) that invites the bots back to the site for repeated crawling and indexing.

3. Feeds:
   If a site has a regularly updated blog or fresh articles posted on it at regular intervals, it would be ideal to have a feed and export it. Google Blog Search and feed tracking help in increasing the crawl activity. When a new post or article is published on the site, the search engine is pinged to let it know that the content has been updated.
4. Importance of Domain:
   A powerful domain that has good quality links coming in from diverse trustworthy domains is very important and it affects both the crawl rate and indexing of the site that resides on that domain.
5. Technical Factors:
   A site can have spider traps in the form of linking structures that have infinite loop system. The crawling can be interrupted by broken links. The problem of duplicate content with same content found on multiple URLs due to use of a CMS is also possible. All these factors inhibit the capacity of a bot to crawl the site exhaustively.
7. Increase the Crawl Rate in Google Webmaster Tools:
   If you login to the Google webmaster tools, there is a provision to increase the spider’s crawl rate. It is a small consolation if the site is affected by problems listed above. On its own, it cannot influence the crawl rate to any extent.

Crawling Factors
You can picture Google (Yahoo and Bing most likely) to consist of a Main Index and a Supplemental Index. The main index consists of the top 10 or 20 results served for important search queries.

If Google thinks a page is not relevant and of high quality, it places it in its Supplemental index. But this index is not visible. A good post by Aaron Wall on Supplemental index will give you a better idea.

The third scenario is where a site’s pages can be crawled and then dropped from the index.

1. Content That is Valuable and Unique:
   To have your pages in the main index, you must provide valuable and unique content. Google is extremely good at identifying content that is unique. Gone are the days when content could be scraped and the introduction and conclusion added to make it look unique. Content that is engaging and valuable definitely play a big part in a page being part of the main index.
2. Domain Importance:
   If a domain has variety of good trustworthy domains pointing to it, it helps Google retain its pages in its main index. A good example is Wikipedia. Some of its pages with just one line of content or duplicate content gets ranked at the top of the SERPs. This is due to the domain trust and authority that Wikipedia commands.
3. PageRank:
   Pagerank or raw link juice is determined by the number of links pointing to your site and their importance. The internal linking structure on your site is also part of the calculation. Pagerank sculpting has become popular over the past few years to direct the link juice to important pages on a site. Links to less perceived important pages are nofollowed. Aaron Wall has remarked that a certain Pagerank threshold is required for a page to be crawled and indexed by Google.
4. External Links:
   If your site has a problem with unique quality content, is on a not so strong domain and does not have enough PR, having a few backlinks from good or lesser known domains to your money pages will be sufficient to get them indexed by Google.

   Oftentimes, I have seen new sites insulated on the web where they have not linked out or have had a lack of incoming links. A simple step such as submitting to a local popular directory that gets crawled by the search engines regularly is enough to get the site pages indexed by Google.

   When the search engines see backlinks to your site, they are literally votes for the site as someone on the web thinks that your site is quite important. That is a key factor that gets your site pages indexed and retained in the index.

5. Other Signals:
   There is a belief among the wider search community that search and traffic volumes to a site, the number of clicks earned by pages on a site on the SERPs, average time spent on a site etc are all signals which search engines are using to retain such pages in their index.

   If the number of visits to a site is increasing steadily and users are spending more time on the site, it is logical to assume that such pages will be found relevant to search queries and retained in the search engines’ index. There is no confirmation officially from the search engines themselves to this effect. There is always the prospect of the signals getting noisy over time.

Rand has done a great video on this in his Whiteboard Friday post on Search Engine Crawling and Indexing factors.

Ravi Venkatesan is a senior SEO consultant at Netconcepts, a well established Auckland SEO company that has a great track record of optimising client sites for organic search and delivering great results.

Yahoo!’s Search Blog announced yesterday that they were making some final changes to their spider, (named “Slurp”), standardizing their crawlers to provide a common DNS signature for identification/authorization purposes.

Previously, Slurp’s requests may have come from IP addresses associated with inktomisearch.com, and now they should all come from IPs associated with domains in this standard syntax:

[something].crawl.yahoo.net

What will this mean to most of us? In most cases, likely nothing. Most sites out there are not likely to be currently performing reverse DNS lookups to check if search engine spiders are actually coming from the IPs/Domains they’re supposed to, except when those spiders get really impolite in requesting too many pages per second. Most people are only identifying bots by their User-Agent strings.

In fact, Yahoo’s provision of this authoritative bot ID syntax is more advanced than Google’s! Google only recommends that people identify their bot (aka “Googlebot”) solely through the User-Agent String — a bit unsatisfactory for a lot of webmasters out there. I’ve heard quite a number of webmasters ask what IP address block to expect the Googlebot requests to originate from, and Google wouldn’t provide them with an authoritative answer.

Of course, one could take a visiting bot’s IP address, say “66.249.65.69″, and perform a Network WHOIS lookup on it to find out if it’s in a block owned by Google. The Network Whois for 66.249.65.69 returns the following info (lookup info provided by Hexillion’s Domain Dossier) :

OrgName: Google Inc.
OrgID: GOGL
Address: 1600 Amphitheatre Parkway
City: Mountain View
StateProv: CA
PostalCode: 94043
Country: US

NetRange: 66.249.64.0 – 66.249.95.255
CIDR: 66.249.64.0/19
NetName: GOOGLE
NetHandle: NET-66-249-64-0-1
Parent: NET-66-0-0-0-0
NetType: Direct Allocation
NameServer: NS1.GOOGLE.COM
NameServer: NS2.GOOGLE.COM
NameServer: NS3.GOOGLE.COM
NameServer: NS4.GOOGLE.COM
Comment:
RegDate: 2004-03-05
Updated: 2007-04-10

OrgTechHandle: ZG39-ARIN
OrgTechName: Google Inc.
OrgTechPhone: +1-650-318-0200
OrgTechEmail: arin-contact

While webmasters could do this lookup for requests for bots displaying the Googlebot user-agent string, it’s still very unsatisfactory because Google does not state that all requests necessarily come from IP blocks that are identifiably owned by Google. So, webmasters would be nervous about blocking something that claimed to be Googlebot yet came from non-Google IP address ranges. After all, it’s possible that Google could have purchased IP addresses and domain names through a proxy in order to perform various types of investigative page requests on sites.

There are cases where hostile dataminers will set their user-agent strings up to masquerade as major search engine spiders, so this newly authoritative method for IDing the bots places Yahoo one step ahead of the game for those webmasters who feel the need to ban the badguys who are scraping their site’s content or requesting pages fast enough to be a defacto denial of service attack.

. . . . . . . . . . . . . . . . . . . .

UPDATE: incrediBILL, one of the moderators at WebmasterWorld, kindly pointed out to me that Matt Cutts had provided the same sort of Googlebot authentication method via the Webmaster Central Blog not long ago. I wish that Google would update their webmaster help section to reflect the same information, if this is indeed intended to be a trustworthy method for authenticating Googlebot. With the instruction only to be found in the blog and not in the actual help section, it still leaves one with the uncomfortable feeling that it’s perhaps an informal method and might still not be depended upon to be true for all cases or it could abruptly change. Hopefully, they’ll update the help pages so everything will be in sync!

Read on to see what I mean.

Sure she’s right in that webmasters don’t have to be afraid if their applications have created multiple page URLs which all contain identical or near-identical content. Websites do this all the time, and search engines aren’t penalizing them for it. (Except perhaps for the case of page-scrapers who steal other sites’ content for redisplay — in which case a scraper’s page might get penalized or just ranked lower as being non-authoritative for its content.) But, webmasters *do* still need to be concerned with duplicate content, because it can affect their overall traffic and rankings.

Quite simply, PageRank continues to be one big factor in ranking one page versus another for keyword searches. Most sites only have so much PageRank to spend on all the pages in their site. If you double the number of pages on your site, you may be virtually cutting each page’s PageRank in half when you do it. If you deploy duplicate copies of all your pages willy-nilly, you’ll have watered-down your pages’ PageRank scores for no good reason.

I wish Jill had mentioned this — dupe content may not cause a website to be penalized, but it’s still an important factor for the sake of improving/optimizing a site’s pages to rank better and bring in more traffic. Her article seems to leave one with the feeling that since there’s not a penalization, webmasters just don’t need to worry about duplication at all.

I say, what do webmasters care if it’s called “penalization” or not, if the end result is still unnecessarily lower rankings in SERPs?                             

If you don’t know what duplicate content may be, you should know that there are a number of things which can cause it to occur in web applications. Primarily, if you have multiple different URLs which all present the same page content, and all of these URLs can be found and indexed by search engine spiders, then you have a duplicate content problem. Here’s some common examples:

• http://example.com & http://www.example.com both present pages for users. As the homepage of your site, if both are indexed by search engines with no qualification, they effectively will split your homepage PR. If subpages are also indexed on both domains, it splits the PR of any of those pages on your site, too. You need one, canonical (i.e. “official”) domain for your site.
    
• http://www.example.com/index.html is the same as http://www.example.com/ — if your site developers linked to the homepage indescriminantly, using both of these types of URLs, they’ve split your homepage’s PageRank.
    
• http://www.example.com/?UID=A7WF5681HJF145I — if your site uses sessionizing in ubiquitous querystrings — assigning session IDs for users for personalization and such — there could be hundreds of different URLs indexed for pages, causing loads of PageRank split through duplication.
    
• www.example.com/app.jsp?DATE=3/17/07&PageID=6 is the same as:
  www.example.com/app.jsp?PageID=6&DATE=3/17/07 and the same as:
  www.example.com/app.jsp?PageID=6&DATE=3/17/07&Link=ad
  - if a page’s URL has multiple querystring terms and different links point into it with the terms in different order, it can create duplication.

These are just some examples — there are many more cases possible.

There are a handful of ways you can fight duplication problems or mitigate their effects on PageRank:

• Fix your site/application so that only one URL per page of content will occur or be found by spiders;
    
• Make the application deliver up NOINDEX metatags for alternate page URLs;
    
• Move user session IDs out of page querystrings and into persistent cookies;
    
• Place 301 redirects on alternative page URLs, redirecting over to the permanant, primary page URL;
    
• If setting up special querystringed page URLs for tracking media campaigns, place those URLs in a special subdirectory on your site which you’ve specified in your robots.txt file for search engines to not index;

There are a number of other solutions out there, depending upon what your duplication problem may be. In most cases, fixing duplication is going to be a bit of a technical clean-up job, but the benefit to your overall page rankings and referral traffic may be significant.

You don’t need to worry about being “penalized” for duplicate content within your site — you’re not going to be delisted for it. But *do* worry about how it affects the SERP rankings for your content.

A group of leading SEOs including Carsten Cumbrowski, Ahmed Bilal, and Rhea Drysdale will review one submission per week delivering a thorough review of usability and site navigation, link building, and copywriting from the perspective of placement in the four leading engines (Google, Yahoo!, MSN and Ask).

It’s clear though that “free” is as free as having your site criticized in one of the SEO clinics experts like to host at conferences.  If chosen for review, the findings and recommendations will be posted for others to peruse.  I’d do as much myself and appreciate their efforts to help others with these case studies but as a website owner, someone responsible for SEO, or marketing manager for a major brand, I might not be so inclined to have my successes and failures outlined in detail for everyone to see.  That concern aside, I do hope they get some quality sites and develop a thorough library of reviews (perhaps I’ll sign up myself!).

To participate, simply contact the team here.

This has a couple of interesting ramifications for SEO professionals and those of us who are optimizing our sites for Google, since bot detection systems may now need to be updated and since this may essentially be a new way of committing site/page submissions into Google’s indices. And we all thought automated URL submissions were dead! I’ll explain further….

First of all, quite a few people like to track the movements of bots through their site pages, in order to know the frequency of spider visits, and to confirm that a page has been spidered, period. For sites/pages which have frequent updates happening upon them, it’s also useful to know the date/time the page gets re-spidered and then to see when the updated text will typically appear in the SERPs. Also, some folks have set their robots.txt to disallow spiders into sections of their sites for various reasons.

So, this change in Google’s spidering functionality will be important for you if you have AdSense ads running on your site. You’ll want to update your robots.txt file to reflect the AdSense bot’s user agent string, and you’ll also want to make sure this user-agent string definition is present in the logfile analysis systems you’re using to track spider activity on your site.

Many of us are using the Web Robots Database at Robotstxt.org in order to identify bots and spiders passing through our sites, and it’s a great resource for all information about the robots exclusion protocol and related matters — Google even cites them as an information resource throughout their webmaster info pages. However, the Robots Database has not been updated to include the definition of the AdSense bot as of the time that I’m writing this. (I’ve just reported the bot identification information over to them to add in, so hopefully this won’t be the case for long.)

(Some webmasters and systems are using the IP address of the bots instead of the User-Agent Strings, but I consider the preferred method to be to use the User Agent for this purpose. Otherwise, you risk counting a search engine employee who is browsing your site during their coffee break to be their spider visiting you!)

Matt Cutts is apparently informally referring to this bot as “Mediabot” or “Media Bot”, but the bot is currently declaring itself with this User-Agent String:

Mediapartners-Google/2.1

If you want to specifically disallow this bot from some section of your site, you should wildcard the bot version number at the end in your robots.txt file like this:

User-agent: Mediapartners-Google*
Disallow: /dont-crawl-this-uri-on-my-site

Another interesting point is raised due to the indexing of the Mediabot-spidered content: doesn’t this basically provide a new way to, errrr, automatically submit pages to Google?!?

If you manually submit your site to Google using their submission form, you’re only allowed to provide the top-level domain name of your site. Using that method, Googlebot will initially visit your homepage, and likely only crawl through one or two levels of links out from the homepage in that initial spidering visit. If you’ve got a really deep site with thousands of pages of content, Googlebot might later revisit the site to try to spider more deeply, in a widening circle out from the homepage. This existing process could utlimately take quite some time before all of your content gets spidered and can begin appearing in the SERPs.

I’m thinking that for cases like that where you have a lot of pages on a new or non-indexed site, adding the Google ads onto all your pages might actually result in them getting initially spidered more rapidly.

Also, quite a lot more pages could potentially get indexed if they have the ads on them, since there are situations where Googlebot will not or cannot spider pages on sites. For instance, if your site content is accessible primarily only through a submission form on your homepage, or through Java/Javascripted menus, or you have only some Flash-enabled navigation system (of course, no SEO professional worth his or her salt would use a Flash-only nav system!) — if your site pages aren’t navigable through regular links displayed on your pages, Googlebot would otherwise never find and index their content.

But, now pages that were only accessible to users through a submission form on your site could potentially now get indexed and appear in SERPs if they have AdSense ads on them!

This is mostly a good thing for those of us working hard to expose content through the SEs, but I bet it could create some havoc for webmasters who are AdSense publishers and who are taken unaware by the potential sudden influx in traffic which pounds their databases as pages suddenly become visible in SERPs. Fun problem to have, though!

Google apparently uses other bots specifically for harvesting other types of content as well. Two that I’ve come across include a bot which grabs images from websites to use in their Google Images section, and a bot for gathering RSS feeds for use in personalized Google homepage, or in the Google Reader.

These are also not identified in Robotstxt.org yet, but their User-Agent strings are as follows:

User-agent: Googlebot-ImageUser-agent: Feedfetcher-Google

Note: Feedfetcher ignores robots.txt exclusion files! Google does this because:

Feedfetcher retrieves feeds only after users have explicitly added them to their Google homepage or Google Reader. Feedfetcher behaves as a direct agent of the human user, not as a robot, so it ignores robots.txt entries. Feedfetcher does have one special advantage, though: because it’s acting as the agent of multiple users, it conserves bandwidth by making requests for common feeds only once for all users.

Do you know if Google is using other specialized bots for other sections of their site or other types of media? If so, I’d be interested in hearing about it.

• The World Wide Web contains 167 terabytes of Web pages on its “surface” (i.e. fixed web pages); in volume this is seventeen times the size of the Library of Congress print collections. Plus another 91,850 terabytes of data in the “deep web” (from database driven websites that create web pages on demand)
• Email generates about 400,000 terabytes of new information each year worldwide.
• The amount of new information stored on paper, film, magnetic, and optical media has about doubled in the last three years.
• Print, film, magnetic, and optical storage media produced about 5 exabytes of new information in 2002. Ninety-two percent of the new information was stored on magnetic media, mostly in hard disks. Five exabytes of information is equivalent in size to the information contained in 37,000 new libraries the size of the Library of Congress book collections.

What I found even more amazing (and depressing) is the degree to which we consume this data. We are a society of
information junkies. Witness this from the same report:

Published studies on media use say that the average American adult uses the telephone 16.17 hours a month, listens to radio 90 hours a month, and watches TV 131 hours a month. About 53% of the U.S. population uses the Internet, averaging 25 hours and 25 minutes a month at home, and 74 hours and 26 minutes a month at work — about 13% of the time.

I can’t imagine sitting in front of the ‘idiot box’ for 131 hours a month. What a terrible waste of one’s life. For an average person, that’s something like 7 years of your life — gone.

Dave of the excellent PassingNotes.com blog looks at it this way:

IF you were all of those things, then of the 720 average hours in a given month, of which you should be sleeping circa 200 (give or take a few hundred), then you’d basically be occupied by media (in some form) for over 330 hours per month – and since we spend about one-third of our lives ‘waiting for something to happen’ (bus, phone etc) and about another 20-40 hours per month in a bathroom (much higher for ted kennedy), then discount sleep, and you’ve got about 80ish hours to be a genuine, sentient human being…sad, sad world…

An example of a page that might look “unfriendly” to a crawler is one that looks like this: http://www.somesite.com/info/default.aspx?view=22&tab=9&pcid=81-A4-76&section=848&origin=msnsearch&cookie=false….URL’s with many (definitely more than 5) query parameters have a very low chance of ever being crawled….If we need to traverse through eight pages on your site before finding leaf pages that nobody but yourself points to, MSNBot might choose not to go that far. This is why many people recommend creating a site map and we would as well.

Why the big deal about index size? From the user’s perspective, a search engine that is comprehensive of the Web in its entirety is going to be more useful than one whose indexation is patchy. Which is why I think the Overture Site Match paid inclusion program from Yahoo! is a really bad idea. Sites shouldn’t pay the search engine to be indexed. Rather, the search engine should strive to index as much of the Web as possible because that makes for a better search engine.

Indeed, I see Google’s announcement as a landmark in the evolution of search engines. Search engine spiders have historically had major problems with “spider traps” — dynamic database-driven websites that serve up identical or nearly identical content at varying URLs (e.g. when there is a session ID in the URL). Alas, search engines couldn’t find their way through this quagmire without severe duplication clogging up their indices. The solution for the search engines was to avoid dynamic sites, to a large degree — or at least to approach them with caution. Over time, however, the sophistication of the spidering and indexing algorithms has improved to the point that search engines (most notably, Google) have been able to successfully index a plethora of previously un-indexed content and minimize the amount of duplication. And thus, the “Invisible Web” begins to shrink. Keep it up, Google and Microsoft!

Ironically, Googlebot regularly gets caught in a spider trap while spidering one of its own sites – the Google Store (where they sell branded caps, shirts, umbrellas, etc.). The URLs of the store are not very search engine friendly: they and are overly complex, and include session IDs. This has resulted in 3,440 duplicate copies of the Accessories page and 3,420 copies of the Office page, for example.

If you have a dynamic, database-driven website and you want to avoid your own site becoming a spider trap, you’ll need to keep your URLs simple. Try to avoid having any ?, &, or = characters in the URLs. And try to keep the number of “parameters” to a minimum. With URLs and search engine friendliness, less is more.