Like any skill though, it takes continual honing and refining, along with the education. This is part of the concept behind the 60-Second Website Audit and training the eye to quickly identify key SEO issues and potential issues.

I’ve joked that, after so many audits, SEO consultants could probably do them blindfolded. So, whip out the blindfold and let’s put that to a test.

Okay, so maybe not really blindfolded, but how about auditing a site without actually seeing the site? Hmm, that might be interesting. This is more than gimmick. In fact, though I might normally take a quick look around a site, ala 60-Second Audit, I generally start the deep dive of an audit exactly as I’m going to show you.

So how do I start auditing a site without looking at the actual site? Just like a search engine, I start at the crawl, which is what we are going to do today. SEO is about so much more than just keywords, and while the title tag is one of the most important signals a site can send to a search engine, a title tag that can’t be found is of little value.

Let’s Audit, Blindfolded

I needed a website I haven’t seen before for this experiment. To pick a site randomly that I haven’t seen, I started by picking a word and searching in Google for it. Being mid-October and feeling the quick approach of winter, the word “sweater” seemed aptly appropriate. I then jumped to the 5th page of results and selected the 2nd result, which was ColdWaterCreek.com. While I know of Cold Water Creek, I don’t recall ever seeing their site, so this is perfect.

Let me introduce you to Xenu. Xenu Link Sleuth™ is a free crawler program that you can download and use to crawl your own site (highly recommend) or other sites. Xenu is one of the most powerful (and perhaps underrated) tools an SEO can have. The amount of information from this tool is priceless, and in this case, will be how we’ll start to audit ColdWaterCreek.com “blindfolded.” That said, if you want to try this at home, I’d recommend doing so on your own site, rather than everyone running out and crawling Cold Water Creek’s site!

Since I prefer to work with the data within Excel, the first thing I do after running it is to export it out to a tab separated format that can be imported into Excel. The details of using Xenu are beyond the scope of this post, so if you are new to Xenu, I highly recommend spending a little time reading first.

After moving the external URLs/links, which in this case also includes the images to another tab, I’m left with 5,161 rows of data, or in other words, 5,161 URLs. When I do a site:www.coldwatercreek.com search in Google (appending &filter=0&start=990 to the URL), I see Google returns 820 results. Quite a difference … what’s the true number?

No idea, but most clients feel that the Google number is often far less than the number of URLs (pages) they have and Xenu seems like a lot more than they expect. I certainly don’t expect Google (or any other search engine) to index 100% of a site’s URLs, but seeing less than 16% indexed based on the Xenu number tells me there may be some issues. Remember though, I haven’t even looked at the site so I don’t really have any idea what those issues are yet.

More importantly, and I see this time after time, we don’t know whether the “true” number should be closer to the Google number or the Xenu number. Too often, people view indexation numbers as something that needs to be increased, that more is better. But more could just as well be indicative of problems.

Next, I note that the server is running Windows IIS as the server platform. I see this in part by the .aspx file extension, but confirmed by the “Server” column of the Xenu report, which lists the URLs as Microsoft-IIS/6.0. In a normal audit, this would clue me in on two things, the first being that I want to be aware of possible case issues. Since IIS doesn’t distinguish case differences, this means that we might see somewhat “sloppy” or inconsistent linking references, such as default.aspx vs. Default.aspx, or blue-jeans.aspx, Blue-Jeans.aspx, Blue-jeans.aspx or even blue-Jeans.aspx. While the server may gladly serve up the appropriate content for these variations, search engine spiders recognize that these could be different pages on other serves, such as Linux, which means that each of these URLs are unique.

Second, this also alerts us to proper redirection challenges. Setting up 301 permanent redirects vs. 302 redirects in IIS seems to trip up many IT departments. So now, we know that this may be a roadblock to any recommendations we make, or that we really need to double check that they have been done in the past and that they are done correctly going forward.

A quick scan is all that is needed to find examples of the case issue:

• http://www.coldwatercreek.com/StoreLocator/Store_Details.aspx?StoreID=9126
• http://www.coldwatercreek.com/Storelocator/Store_Details.aspx?StoreID=9126

• http://www.coldwatercreek.com/StoreLocator/Store_Events.aspx?StoreID=146
• http://www.coldwatercreek.com/Storelocator/Store_Events.aspx?StoreID=146

See the differences above? One version uses an upper case “L” while the other version uses a lower case “l” within StoreLocator. This means that there are likely duplicates of all of the stores for the details and events pages. Fortunately, the rest of the URLs are pretty clean and consistent, which isn’t always the case (no pun intended).

After sorting by URL, we quickly see some other common issues. Ironically, this issue is less about the content that is there and more about what isn’t there. Toward the top of the list, we find these URLs:

• http://www.coldwatercreek.com/%2f404.htm
• http://www.coldwatercreek.com/404.htm
• http://www.coldwatercreek.com/404.htm?aspxerrorpath=/MyAccount/MyAccount/MyAcctLogin.aspx

First, the %2f is the encoding for the “/” which tells me that there is probably a malformed URL here. The real issue though is that all of these represent a 404 file not found page, yet all return a 200 ok header status. This means that these, and likely any malformed URLs or URLs that no longer exist, will continue to live on and bloat the index rather than drop out. It also means that the site isn’t sending the highest quality signal to search engines by appearing to return ok statuses for URLs that don’t exist. However, we can also see by the following example that some URLs are returning a proper 404 header status:

• http://www.coldwatercreek.com/GiftCard/%2fGiftCard%2fEGiftCardATB.aspx%3fproductid%3d01GC006%26ensembleid%3d10756

Along with additional examples of possible encoding issues, this is a good reminder that we need to check for issues in different areas of the site and in different ways. This is especially true with complex sites that might appear seamless visually, but may be powered by a number of different scripts, such as a content management system, ecommerce cart, blog, forum and FAQ … all of which might be powered separately.

While we are on that note, my guess is that the following URL probably isn’t of much value either:

• http://www.coldwatercreek.com/Blank.htm

Other examples of possible duplication or diluted content may be seen in the following:

• http://www.coldwatercreek.com/default.aspx (because I still haven’t viewed the site, I don’t know for certain, but experience and my gut tells me that this may be a duplication of the homepage URL, http://www.coldwatercreek.com/).

Another challenging area for duplication is in presentation, and my guess is that:

• http://www.coldwatercreek.com/apparel/pants/longs.aspx

is probably being duplicated by:

• http://www.coldwatercreek.com/apparel/pants/longs.aspx?ShowAllProducts=false

Which is prevalent for all products that have pagination, and to some extent, duplicated further by the counter URL variation:

• http://www.coldwatercreek.com/apparel/pants/longs.aspx?ShowAllProducts=true

At this point, we don’t know if these are being dealt with in other ways, such as robots handling, nofollow link attributes or the canonical link element. Nor can we determine how these should be handled, but at least we have a better understanding of what is going on and what we need to dig into.

Related to that, we also have a quick view into the pagination URL construct:

• http://www.coldwatercreek.com/apparel/pants/longs.aspx?page=1

• http://www.coldwatercreek.com/apparel/pants/longs.aspx?page=2

In addition, some other URLs that are probably low value for search (as well as possible encoding issues again) that we’ll want to check to see whether they are being excluded from the bots:

• http://www.coldwatercreek.com/GiftCard/%2fGiftCard%2fGiftCardATB.aspx%3fproductid%3d44130%26ensembleid%3d50183

• http://www.coldwatercreek.com/GiftCard/EGiftCard.aspx?productid=01GC001&ensembleid=10756

While we are looking at URLs, we can quickly scan our Excel file and identify key URL constructs being used.

Such as parameter-based product detail pages, including perhaps low-value duplication:

• http://www.coldwatercreek.com/Products/Detail.aspx?productid=30144&ensembleid=34281

• http://www.coldwatercreek.com/Products/Detail.aspx?productid=30144&ensembleid=34281&Skn=outlet

Some lengthy, a little parameter heavy and generally ugly URLs that may be challenging to bots:

• http://www.coldwatercreek.com/Products/prodList.aspx?provider=productsearch&cmd=czcategory&cat=All+Products////Apparel////Dresses////UserSearch=Dept.Channel+ID=1&ShowAllProducts=false

• http://www.coldwatercreek.com/Sale/default.aspx?provider=productsearch&cmd=czNewPage&path=All+Products////UserSearch=RedTag////UserSearch=Misses&page=18

Especially in comparison to the cleaner, keyword friendly category pages (though I’d still want to review these around the pagination construct mentioned earlier):

• http://www.coldwatercreek.com/knit-dresses.aspx

• http://www.coldwatercreek.com/misses.aspx

• http://www.coldwatercreek.com/outlet-jackets.aspx

Seem like a lot of focus on URLs? You bet. URLs are at the foundation of a site’s SEO. Get these wrong and little else matters. Xenu is excellent at seeing the URLs that exist that may not ever make it into a search engine index … which may be quite telling. So remember that a “site:domain.com” advanced query only reveals what is “above water.”

The Cold Water Creek site is actually quite small with only a few variations of URLs. The real power and beauty of Xenu is when you find yourself reviewing 300,000+ URLs from highly complex sites with several URL constructs and variants.

In part 2 of the Blindfolded SEO Audit, we’ll start out by seeing what Xenu can show us about the most important search signal a site has.

Back in 2004, I laughed and laughed and laughed, and I told coworkers that it was a huge waste of money, because, I said, “they won’t be able to buy themselves into the top position for searches for ‘Yellow Pages’”. SuperPages.com long held that distinction under my SEO direction, and I knew that purchasing the term in a domain name alone would not depose all the work we’d done to rank tops for it. As time passed, however, yellowpages.com has indeed deposed the Superpages forerunner.

Still, the domain name didn’t accomplish this alone. There were a mixture of factors involved. AT&T used a considerable war chest of new capital to gain marketshare in the YP space. I believe they likely undercut other players and made major traffic deals with partners in order to gain greater traffic — so, some of their position was purchased. Also, the AT&T company has much greater prestige and far more customers than the previous owners of the “yellowpages.com” domain name, and they could leverage their print phone book covers to further promote the URL. This meant that far more people were inclined to link to them, and the domain name itself, embedded in all those inlinks, gave AT&T a defacto advantage in the PageRank game.

Further, Google itself shifted ranking factors slightly between now and 2004, placing a bit greater weight upon keyword terms within domain names.

And, after I left the Superpages, a number of my recommendations for ongoing SEO were neglected to some degree by the people overseeing such work. This resulted in Superpages getting deposed from its longtime slot as first position for “yellow pages”, and Yellowpages.com has been occupying that position for a while since.

Just for the sake of egos and prestige value, it might appear on the surface that AT&T’s investment in the yellowpages.com domain name was justified in order for them to achieve top ranking. But, this is the world of business, and considering the direction in which society and commerce has been shifting, their purchase of that old domain name and the more recent purchase of YP.com appear to be at a hyper-inflated cost that is way out of whack with any good business rationale.

First, I know for a fact that they never have gotten sufficient value from that original $100 million dollar domain name. Knowing exactly how much traffic goes to the top position for “yellow pages”, I know that it’s never paid for itself, and it’s unlikely to do so. (Okay, so I know that they also got a number of other business assets along with the domain in that original $100 mill price-tag, but even considering that, this domain was a very bad buy.)

I’ve written previously about the decline in user searches for “yellow pages”, and that decline is continuing:

(Google Trends graph shows ongoing erosion of user searches for “yellow pages”)

Fewer and fewer users are typing “yellow pages” into Google, as this graph demonstrates. It would’ve taken “yellowpages.com” a good many years to repay for itself even when the traffic for “yellow pages” keyword searches was good, but with this declining trend in those searches very apparent, this is no longer a realistic longterm investment, and it will turn more rapidly into a major capital loss.

Considering this ongoing, downward trend, the monetary worth and traffic-bearing value of both “yellowpages.com” and “yp.com” are declining. Many feel that the overall concept of “yellow pages” could be fast disappearing as well, and that will leave these as very dated-feeling trademarks only a short ways down the road. I’d say there are a very good number of young consumers today who don’t even know what “yellow pages” is! So, even the brand name value of these appears very short-lived.

I’m doubting that even the short term, ostensible existing traffic of YP.com is worth this $3.85 million price tag.

If I were an AT&T stockholder, I think I’d be mad enough about this to demand an accounting. With a number of major yellow pages companies experiencing significant business problems currently, this purchase is just incomprehensible. But, since AT&T’s yellow pages business unit is still attached to their telephone company, perhaps they have the luxury of wasteful spending on this piece.

Can you keep a secret? There is an unfair advantage. Amazon is applying conditional 301 URL redirects through their massive affiliate marketing program.

Most online merchants outsource the management and administration of their affiliate program to a provider who tracks all affiliate activity, using special tracking URLs. These URLs typically break the link association between affiliate and merchant site pages. As a result, most natural search traffic comes from brand related keywords, as opposed to long tail keywords. Most merchants can only imagine the sudden natural search boost they’d get from their tens of thousands of existing affiliate sites deeply linking to their website pages with great anchor text. But not Amazon!

Amazon’s affiliate (“associate”) program is fully integrated into the website. So the URL that you get by clicking from Guy Kawasaki’s blog for example to buy one of his favorite books from Amazon doesn’t route you through a third party tracking URL, as would be the case with most merchant affilate programs. Instead, you’ll find it links to an Amazon.com URL (to be precise: http://www.amazon.com/exec/obidos/ASIN/0060521996/guykawasakico-20), with the notable associate’s name at the end of the URL so Guy can earn his commission.

However, refresh that page with your browser’s Googlebot User Agent detection turned on, and you’ll see what Googlebot (and others) get when they request that same URL: http://www.amazon.com/Innovators-Dilemma-Revolutionary-Business-Essentials/dp/0060521996 delivered via a 301 redirect script. That’s the same URL that shows up in Google when you search for this book title.

So if you are a human coming in from affiliate land, you get one URL used to track your referrer’s commission. If you are a bot visiting this URL, you are told these URLs now redirect to the keyword URLs. In this way, Amazon is able to have its cake and eat it too – provide an owned and operated affiliate management system while harvesting the PageRank from millions of deep affiliate backlinks to maximize their ranking visibility in your long tail search query.

(Note I’ve abstained from hyperlinking these URLs so bots crawling this content do not further entrench Amazon’s ranking on these URLs, although they are already #4 in the query above!).

So is this strategy ethical? Conditional redirects are a no-no because it sends mixed signals to the engine – is the URL permanently moved or not? If it is, but only for bots, then you are crossing the SEO line. But in Amazon’s case it appears searchers as well as general site users also get the keyword URL, so it is merely the affiliate users that get an “old” URL. If that’s the case across the board, it would be difficult to argue Amazon is abusing this concept, but rather have cleverly engineered a solution to a visibility problem that other merchants would replicate if they could. In fact, from a searcher perspective, were it not for Amazon, many long tail product queries consumers conduct would return zero recognizable retail brands to buy from, with all due respect to PriceGrabber, DealTime, BizRate, NexTag, and eBay.

As a result of this long tail strategy, I’d speculate that Amazon’s natural search keyword traffic distribution looks more like 40/60 brand to non-brand, rather than the typical 80/20 or 90/10 distribution curve most merchants (who lack affiliate search benefits) receive.

Brian

This might confuse search marketers about the value of providing RSS feeds on one’s site for the purposes of natural search marketing. Here at Netconcepts, we’ve recommended using RSS for retail sites and blogs for quite some time, and we continue to do so. Webmasters often take syndicated feeds in order to provide helpful content and utilities on their sites, and so providing feeds can help you to gain external links pointing back to your site when webmasters display your feed content on their pages.

Google has removed RSS feed content from their regular SERPs, but they haven’t necessarily reduced any of the benefit of the links produced when those feeds are adopted and displayed on other sites. When RSS and Atom feeds are used by developers, they pull in the feed content and then typically redisplay it on their site pages in regular HTML formatting. When those pages link back to you as many feed-displayed pages do, the links transfer PageRank back to the site originating the feeds, and this results in building up ranking values.

So, don’t stop using RSS or Atom feeds!

Not long ago, I wrote an article on how some local directory sites were using subdomains in an attempt to achieve good ranking results in search engines. In that article, I concluded that most of these sites were ranking well for other reasons not directly related to the presence of the keyword as a subdomain — I showed some examples of sites which ranked equally well or better in many cases where the keyword was a part of the URI as opposed to the subdomain. So, in Google, subdirectories were already functioning just as well as subdomains for the purposes of keyword rank optimization.

I’ve seen a lot of sites which had varying degrees of quality in their subdomaining strategies. If you do have subdomains, you should ideally insure that they contain primarily unique content not reflected on your other domains — each subdomain should contain page content that does not also live on other subdomains or else it can appear that you are attempting to spam the search engine indices.

Google’s Webmaster Guidelines are very clear on this subject:

“Don’t create multiple pages, subdomains, or domains with substantially duplicate content.”

Most large corporate websites have some level of accidental duplicate content, but if you deploy dozens or hundreds of subdomains with all dupe text, it will appear that you’re purposefully trying to spam the search engines — don’t do it.

If you are considering how to structure your URLs and site content for natural search marketing, I’d say you might be better off just using a simple format of descriptively keyworded directories and subdirectories rather than keyworded subdomains. This is often easier to manage, and it looks a lot more natural/reasonable from the search engines’ perspective. There’s lower likelihood of accidentally mirroring/duplicating your content, too.

Don’t freak out if you have a few subdomains — this is also natural. Many major websites host different site sections and applications on subdomains, and some have external providers delivering content on separate servers — it’s very easy in those cases to assign a subdomain to the third party that’s providing service for you. As long as you’re not duplicating the main content of your pages on the subdomains, this is fine.
Finally, I’ve had a number of people ask my opinion regarding foreign languages — which is better, subdomain or subdirectory.

I actually prefer using separate top-level domains (“TLDs”) for this purpose, since it allows you to send a very clear signal to the search engines that particular content is intended for various countries. For instance, your French language pages could be delivered on .FR domains like: www.example.fr

However, if for some reason you don’t wish to use foreign TLDs for your alternate language pages, you should not worry overly about using separate subdomains versus directory/subdirectories. “french.example.com” will likely function just as well as “www.example.com/french/” in my opinion. I believe that translated versions of pages are NOT counted as duplicate content because they essentially contain very different text. Yes, the information may be duplicated, but the text content is not, and pages in two different languages are far less likely to both come us as relevent for the same keyword search.

So, for foreign language pages, I recommend separate TLDs for best performance, or else use whatever approach is easiest for you to set up and maintain.

Maile suggests a few ways of addressing dupe content, and she also reveals a few details of Google’s workings that are interesting, including:

• Unnecessarily long URLs are unattractive, and might reduce chances that a user would click through to your page. While this sounds like a subjective opinion that’s a bit counter-intuitive (because one could assume that users focus more on link titles than length of URLs), it’s quite possible that Google would have done enough experimentation during their usability testing to know for sure that longer URLs might actually have a negative impact on click-through rates. So, avoid longer URLs if possible in your application design!
• When deciding what to display from your site for a user’s search, if Google detects duplicate content matching the user’s query, they’ll group all the dupe pages into a cluster, and then apply some methods to choose which of your pages would be the best choice to present to their searcher.
• They attempt to focus the collective “link popularity” or PageRank from all members of a cluster on your site to one page. This is slightly odd, since it runs counter to her earlier statement that duplication can cause “link popularity” dilution. Likely, this means that there are cases when Google can find it difficult to cluster all dupes from a site, so it’s still best to reduce duplication rather than solely rely upon their algorithms to handle it for you.
• She suggests using a Sitemap to inform them of the primary URLs of a site, which suggests that Google may be using the sitemaps as a prime indicator when trying to select a canonical URL for a particular cluster. I note that while using a sitemap could help Google select which of your dupe pages to present to a user, it doesn’t really solve your entire dupe problem — you should still try to use additional methods to manage dupes.

Maile’s suggestions reiterate some of the de-duplication advice I’ve previously given, and I’ve also suggested having your site resolve to a single domain name to reduce duplication (along with Google’s Matt Cutts, who also recommends domain canonicalization).

Each of the search engines handle duplication issues a little differently, making it desirable to use best practices to manage the issue, if you really want to improve your site’s natural search performance.

I’ve previously opined that classic SEO may become extinct in favor of Usability, and announcements like this fluid handling of underscores would tend to support that premise. Google, Yahoo! and MS Live Search have been actively trying to reduce barriers to indexation and ranking abilities by changes like this plus improved handling of redirection, and myriad other changes which both obviate the need for technical optimizers and reduce the ability to artificially influence rankings through technical improvements.

I continue to think that the need for SEOs may decrease until they’re perhaps no longer necessary, so natural search marketing shops will likely evolve into site-building/design studios, copy writing teams, and usability research firms. The real question would be: how soon will it happen?

What is that, you might ask? It’s a simple proposal for sites to do away with using the WWW-dot-domainname format for URLs, and to instead go with the non-WWW version of domains instead. Managing your site’s main domain/subdomain name is one basic piece of search engine optimization, and this initiative can be a guide for how to decide which domain name will become the dominant one for a site. Read on for more info…

I think there are strong arguments in favor of dropping the “dub-dub-dub” because use of a domain with it is not really necessary. Most sites are set up so that either primary domain name will resolve to the same pages.

http://www.example.com/

http://example.com/

Removal of the WWW can help streamline pagecode which can reduce data transfer and speed up page load times. (Admittedly, the removal of the WWWs only improves data transfer and page load by relatively insignificant amounts in the majority of cases, but such seemingly minor changes could have very large impacts for large-scale sites which deliver millions of pages per day, and the cumulative reduction of data transferred might even result in some tiny reduction in one’s ISP hosting service costs.) And, it is visually cleaner to lose the WWW when citing a company’s URL in print and other marketing collateral where the domain or URL needs to be displayed.

Since the major search engines have been handling both types of URLs relatively consistently, I think it might be useful to use the No-WWW philosophy when you work on canonicalizing your WWW/No-WWW domain names. As you may be aware, many sites have both the WWW and non-WWW versions of their URLs in play simultaneously. For purposes of SEO, one should have one redirect to the other so that you have one primary, or “canonical” version of your main domain, and all your PageRank can be applied to pages delivered on that domain instead of being spread thinner across both versions. But, which version — with WWW, or without WWW — should be your primary? The No-WWW initiative provides a good reason to choose one above the other.

If you’re considering joining this effort, there’s no reason to do this in a costly manner. Your IT support personnel can typically set up redirection with relatively minor effort, so beginning the conversion process to drop the WWWs from all page code and application interfaces doesn’t have to be done in sync. Using 301 redirects will work for SEO and insure that all the legacy URLs will continue to work for users. Then, over time, convert the URLs in your interfaces as that code is touched when working on other projects.

The WWW subdomains are unnecessary and should be deprecated. Join the effort to simplify!

• CitySearch
• Craigslist
• Local.com

In that installment, I conclude that the subdomaining for the sake of keyword ranking has no real benefit.

This assertion really can be extended out to all other types of sites as well, since the ranking criteria that the search engines use is not limited to only local info sites. Keywords in subdomains really have no major benefit.

SEO firms used to suggest that people deploy their content out onto “microsites” for all their keywords – a different domain name to target each one. This just isn’t a good strategy, really. Focus on improving the quality of content for each keyword, founded on its own page, and work on your link-building efforts (quality link-building, not unqualified bad-quality links). Tons of keyword domains or subdomains is no quick solution for ranking well.

Read on to see what I mean.

Sure she’s right in that webmasters don’t have to be afraid if their applications have created multiple page URLs which all contain identical or near-identical content. Websites do this all the time, and search engines aren’t penalizing them for it. (Except perhaps for the case of page-scrapers who steal other sites’ content for redisplay — in which case a scraper’s page might get penalized or just ranked lower as being non-authoritative for its content.) But, webmasters *do* still need to be concerned with duplicate content, because it can affect their overall traffic and rankings.

Quite simply, PageRank continues to be one big factor in ranking one page versus another for keyword searches. Most sites only have so much PageRank to spend on all the pages in their site. If you double the number of pages on your site, you may be virtually cutting each page’s PageRank in half when you do it. If you deploy duplicate copies of all your pages willy-nilly, you’ll have watered-down your pages’ PageRank scores for no good reason.

I wish Jill had mentioned this — dupe content may not cause a website to be penalized, but it’s still an important factor for the sake of improving/optimizing a site’s pages to rank better and bring in more traffic. Her article seems to leave one with the feeling that since there’s not a penalization, webmasters just don’t need to worry about duplication at all.

I say, what do webmasters care if it’s called “penalization” or not, if the end result is still unnecessarily lower rankings in SERPs?                             

If you don’t know what duplicate content may be, you should know that there are a number of things which can cause it to occur in web applications. Primarily, if you have multiple different URLs which all present the same page content, and all of these URLs can be found and indexed by search engine spiders, then you have a duplicate content problem. Here’s some common examples:

• http://example.com & http://www.example.com both present pages for users. As the homepage of your site, if both are indexed by search engines with no qualification, they effectively will split your homepage PR. If subpages are also indexed on both domains, it splits the PR of any of those pages on your site, too. You need one, canonical (i.e. “official”) domain for your site.
    
• http://www.example.com/index.html is the same as http://www.example.com/ — if your site developers linked to the homepage indescriminantly, using both of these types of URLs, they’ve split your homepage’s PageRank.
    
• http://www.example.com/?UID=A7WF5681HJF145I — if your site uses sessionizing in ubiquitous querystrings — assigning session IDs for users for personalization and such — there could be hundreds of different URLs indexed for pages, causing loads of PageRank split through duplication.
    
• www.example.com/app.jsp?DATE=3/17/07&PageID=6 is the same as:
  www.example.com/app.jsp?PageID=6&DATE=3/17/07 and the same as:
  www.example.com/app.jsp?PageID=6&DATE=3/17/07&Link=ad
  - if a page’s URL has multiple querystring terms and different links point into it with the terms in different order, it can create duplication.

These are just some examples — there are many more cases possible.

There are a handful of ways you can fight duplication problems or mitigate their effects on PageRank:

• Fix your site/application so that only one URL per page of content will occur or be found by spiders;
    
• Make the application deliver up NOINDEX metatags for alternate page URLs;
    
• Move user session IDs out of page querystrings and into persistent cookies;
    
• Place 301 redirects on alternative page URLs, redirecting over to the permanant, primary page URL;
    
• If setting up special querystringed page URLs for tracking media campaigns, place those URLs in a special subdirectory on your site which you’ve specified in your robots.txt file for search engines to not index;

There are a number of other solutions out there, depending upon what your duplication problem may be. In most cases, fixing duplication is going to be a bit of a technical clean-up job, but the benefit to your overall page rankings and referral traffic may be significant.

You don’t need to worry about being “penalized” for duplicate content within your site — you’re not going to be delisted for it. But *do* worry about how it affects the SERP rankings for your content.