Yahoo! Local’s Frazier Miller

It was very interesting to hear the take on local & mobile from one of Yahoo! Local’s top thought leaders. It was obvious that Frazier has a very tight grip on understanding what motivates consumers and where the trends may be headed in local/mobile evolution.

Some highlights of Frazier’s presentation included:

• Mention of how Yahoo Answers has incorporated a local component;
• How SearchMonkey could provide more compelling listings in SERPs;
• Need of users to be able to easily search government sites;
• Heavy interest at Yahoo in Events, which is why they purchased Upcoming in late ’05;
• They consider Local Product Search to be the “holy grail” of local search;
• The convergence of Local and Social is compelling, although Yahoo feels it’s not yet been adequately addressed and there are needs for managing privacy considerations in the way that it ultimately gets implemented;
• The “hyper-local” trend is perhaps getting filled out by a number of niche players;
• Geotargeting will provide the next acceleration of ROI in advertising;
• Educated the audience on the “ROBO” acronym: “research online, buy offline”;

Frazier’s candidness was refreshing when he spoke openly about other companies’ services which he found to be highly compelling for local work. These included:

• Walkscore.com – a service which helps users decide how difficult a walking route could be — something that’s highly useful to many during these gas-crunch times.
• Yelp – the highly-popular business rating and directory site;
• Craigslist – the well-known classifieds site;
• Facebook – the growing social networking site;

What Yahoo! may be working on for the future:

• Better geotargeting capabilities for advertising;
• Expanding Ad Exchanges and Ad Networks – “making the local ad market”;
• Increasing merchant awareness — channel resellers are transforming sales teams to educate merchants about online marketing. Some of these include major IYPs such as Yellowpages.com, Yellowbook.com, and Idearc Superpages.com.

Frazier stated that there are reasons to believe that mobile search marketing may finally be arriving, despite past predictions of critical mass that were off-base. The devices and hardware have evolved, with Moore’s law reducing down the cost of the products. Connectivity rates and badwith have improved. And devices and carriers are both opening up for app developers, with the technology getting beyond just SMS.

Yahoo’s Blueprint platform for mobile is different from Google’s Android, and may be superior in terms of easing development. Yahoo’s motto for this is “write once, run anywhere”.

Based on Frazier’s presentation, I think it will be interesting to continue to watch what Yahoo! is developing and deploying in the Local Search space.

This deal will help make Yahoo! Search results far safer for innocent users clicking through to new sites, and it will likely enhance the comprehensiveness of McAfee security products for all of their software users. Viruses, spyware and other forms of malware are very often tied to internet sites. Some webpages are little more than Trojan Horses, for instance, pretending to be a reputable site, and attempting to lure the unwary into typing their passwords in for banking, credit cards, eBay, etc — all so that the unethical people responsible may steal identities, empty out bank accounts, and charge up credit cards fraudulently. And, there are many other types of exploits tied to internet locations, including sites which start to download invasive software and viruses into one’s computer through the browser interface.

The Yahoo/McAfee partnership appears to me to be very valuable to both parties and to the public.

The deal is also likely to provide something of a cost savings to Yahoo!, I would guess, since it could pave the way towards reducing their costs of working to identify malware sites. McAfee is an industry leader in this sector, and contracting for their services could allow Yahoo to not have to duplicate so many security and scanning types of functions.

The Blogoscoped mention of Yahoo’s love for Flickr content is particularly timely, since Yahoo! announced back in June that they were permanently shutting down Yahoo! Photos in favor of their Flickr property, and the final closing date is tomorrow, September 20th.

Previously, I’d railed a bit against Yahoo! because I’d seen a lot of evidence that they didn’t spider/index Flickr content as well or comprehensively as Google did — altogether ironic since Yahoo owns Flickr. Just as with the anecdotal reports in the Blogoscoped post, I’m seeing nice indications that my earlier criticism of Yahoo’s lack of inclusion of Flickr content may now be completely resolved.

For instance, for experimentation purposes, I optimized a number of pictures of the picturesque Holly Hill House on Catalina Island via Flickr over a year ago. For many months, none of those pictures were showing up at all in Yahoo’s Image Search results (while, they were indexed and ranking really well in Google Image Search results within just a few weeks). Now, TWELVE of my Holly Hill House pics are appearing in the first 20 image search results on Yahoo:

(click to enlarge)

So, I’d say there is now even more benefit to optimizing through Flickr, since there’s greater chance of getting search referrals from Yahoo! Image Search results.

So far, I’m not actually seeing more referral traffic from Yahoo because of this, though, but I’ll soon be deploying much larger sample sets for the purposes of experimentation, and I’ll circle back around to report my statistical findings.

I can still critique Yahoo! a bit, though: images newly uploaded to Flickr seem to still have a significant amount of time before they get included in Yahoo Image Search results. Why? These images show up really rapidly via search in Flickr, so one assumes Yahoo would be able to pull those contents into their SERPs through a federated search of Flickr… The images really should be available quicker, and Yahoo should improve absorption of new Flickr images content faster.

I’ve previously opined that classic SEO may become extinct in favor of Usability, and announcements like this fluid handling of underscores would tend to support that premise. Google, Yahoo! and MS Live Search have been actively trying to reduce barriers to indexation and ranking abilities by changes like this plus improved handling of redirection, and myriad other changes which both obviate the need for technical optimizers and reduce the ability to artificially influence rankings through technical improvements.

I continue to think that the need for SEOs may decrease until they’re perhaps no longer necessary, so natural search marketing shops will likely evolve into site-building/design studios, copy writing teams, and usability research firms. The real question would be: how soon will it happen?

Yahoo!’s Search Blog announced yesterday that they were making some final changes to their spider, (named “Slurp”), standardizing their crawlers to provide a common DNS signature for identification/authorization purposes.

Previously, Slurp’s requests may have come from IP addresses associated with inktomisearch.com, and now they should all come from IPs associated with domains in this standard syntax:

[something].crawl.yahoo.net

What will this mean to most of us? In most cases, likely nothing. Most sites out there are not likely to be currently performing reverse DNS lookups to check if search engine spiders are actually coming from the IPs/Domains they’re supposed to, except when those spiders get really impolite in requesting too many pages per second. Most people are only identifying bots by their User-Agent strings.

In fact, Yahoo’s provision of this authoritative bot ID syntax is more advanced than Google’s! Google only recommends that people identify their bot (aka “Googlebot”) solely through the User-Agent String — a bit unsatisfactory for a lot of webmasters out there. I’ve heard quite a number of webmasters ask what IP address block to expect the Googlebot requests to originate from, and Google wouldn’t provide them with an authoritative answer.

Of course, one could take a visiting bot’s IP address, say “66.249.65.69″, and perform a Network WHOIS lookup on it to find out if it’s in a block owned by Google. The Network Whois for 66.249.65.69 returns the following info (lookup info provided by Hexillion’s Domain Dossier) :

OrgName: Google Inc.
OrgID: GOGL
Address: 1600 Amphitheatre Parkway
City: Mountain View
StateProv: CA
PostalCode: 94043
Country: US

NetRange: 66.249.64.0 – 66.249.95.255
CIDR: 66.249.64.0/19
NetName: GOOGLE
NetHandle: NET-66-249-64-0-1
Parent: NET-66-0-0-0-0
NetType: Direct Allocation
NameServer: NS1.GOOGLE.COM
NameServer: NS2.GOOGLE.COM
NameServer: NS3.GOOGLE.COM
NameServer: NS4.GOOGLE.COM
Comment:
RegDate: 2004-03-05
Updated: 2007-04-10

OrgTechHandle: ZG39-ARIN
OrgTechName: Google Inc.
OrgTechPhone: +1-650-318-0200
OrgTechEmail: arin-contact

While webmasters could do this lookup for requests for bots displaying the Googlebot user-agent string, it’s still very unsatisfactory because Google does not state that all requests necessarily come from IP blocks that are identifiably owned by Google. So, webmasters would be nervous about blocking something that claimed to be Googlebot yet came from non-Google IP address ranges. After all, it’s possible that Google could have purchased IP addresses and domain names through a proxy in order to perform various types of investigative page requests on sites.

There are cases where hostile dataminers will set their user-agent strings up to masquerade as major search engine spiders, so this newly authoritative method for IDing the bots places Yahoo one step ahead of the game for those webmasters who feel the need to ban the badguys who are scraping their site’s content or requesting pages fast enough to be a defacto denial of service attack.

. . . . . . . . . . . . . . . . . . . .

UPDATE: incrediBILL, one of the moderators at WebmasterWorld, kindly pointed out to me that Matt Cutts had provided the same sort of Googlebot authentication method via the Webmaster Central Blog not long ago. I wish that Google would update their webmaster help section to reflect the same information, if this is indeed intended to be a trustworthy method for authenticating Googlebot. With the instruction only to be found in the blog and not in the actual help section, it still leaves one with the uncomfortable feeling that it’s perhaps an informal method and might still not be depended upon to be true for all cases or it could abruptly change. Hopefully, they’ll update the help pages so everything will be in sync!

To support this initiative, they will jointly support sitemaps.org. If you recall, “sitemaps” was the product name that Google had been using, and which became deprecated just a few months ago in favor of “Google Webmaster Tools”. Obviously, the wheels had already begun turning to repurpose the “Sitemaps” brand name into a jointly-operated service.

Now when Sitemaps are generated to follow the common protocol, webmasters will still need to submit the link feeds to each of the SEs via their existing managment tools such as in Google Webmaster Tools and in Yahoo! Site Explorer.

If you recall, I was one of a number of webmasters out there who had requested that they collaborate on a common protocol, such as in a blog post I wrote back in September:

“Hopefully each of the major search engines will try to employ identical or compatible formats for site URLs, because it will be a hassle to have to keep up with multiple formats. This is an area where the SEs really ought to cooperate with one another for “pro bono publicoâ€? – for the common good. Currently, Yahoo seems to be just defensively immitating Google in this arena, and no one’s showing signs of collaborating.”

Kudos to Google and Yahoo for overcoming traditional corporate competitiveness to do something that mutually benefits website owners as well as the search engines!

 

So, what is my evidence that Google’s development of Quality criteria is becoming more influential in their rankings than PageRank and other classic optimization elements? Read on and I’ll explain. Nearly since inception, Google has focused strongly upon Usability and User Experience to create a service that Marissa Mayer (Google’s Vice President of Search Products & User Experience), has referred to as a useful tool which they made approachable/accessible by applying a usable interface on top of it. We know that they haven’t stopped working to improve the user experience on their site, but are following the practice of continuous quality improvement in this area. Way back in 2002 Mayer reported:

“we’re user testing almost every week. We’ll do a site-wide test once a month or so, with some tasks, but more free-form, just to see where people go, where they encounter problems. The other three weeks of the month, we test specific features.”

You can safely bet that they’ve increased or evolved their user testing methods here in 2006. Google has been hiring Quality Evaluators, Quality Raters, and Usability Researchers at an astonishing pace during this past year.

In 2005, Henk van Ess reported various details of instructions that Google provides to their human evaluators to use in rating the quality of pages appearing in their search results. His first two entries on the subject were eye-openers for many:

“Google’s Secret Lab, Prelude”
“Google’s Human Quality Evaluation: How To Spot Offensive Sites & Google’s Whitelist“

Google, Yahoo, and MSN all use some human evaluators in their ongoing fight against spam sites. John Battelle relates in his book, The Search (2005) pg 240, that Yahoo exercises editorial discretion to customize some types of content for various keyword SERPs. He also wrote, “Google sees the problem as one that can be solved mainly through technology — clever algorithms and sheer computational horsepower will prevail. Humans enter the search picture only when algorithms fail — and then only grudgingly.” Yet, Google more than any other has apparently expanded the role of humans in evaluating their search results, and their role has gone beyond just red-flagging spam sites. They are apparently rating pages based upon quality and appropriateness for the search keyword as well.

Wall Street reports that Google recently changed their internal focus from deploying more products to improving core services and making them integrate better with each other. You can read the subtext here: improve the quality of their core products, and their central product is search.
They’ve also turned attention to the quality of pages that the ads running on their network are linked. Jeremy Shoemacher (aka “Shoemoney”) and others have only just reported that Google is now rating the Quality of AdWords landing pages. (Read the AdWords blog entry on the subject, too.) This is another indicator of their dedication to the quality of user experience — they’re essentially penalizing their advertisers who have lower quality.

So, there’s plenty of evidence that Google is now obsessed with taking the quality of their SERPs to the next level by testing to see if the pages appearing in results are apropos, and then altering the rankings of the results to better target user’s desires/intentions. How do they do this?

You can read the documents exposed on Henk’s Search Bistro site for specifics, but I can summarize a bit:

• Their evaluators are presented the search results page for a keyword. They must check out each of the top pages appearing for that keyword term, and vote on whether the page is good quality or not.
• If the page content seems inappropriate for the keyword, it will get a bad rating for that keyword.
• They particularly will give a bad rating to pages which are primarily composed of ads.
• They particularly will give a bad rating to pages which are primarily composed of affiliate links, unless the site has included some significant value-add content in conjunction with the affiliate content.
• Google desires that the pages linked in their SERP for a keyword should not all contain identical content, because they believe this would be a bad user-experience. So, syndicated content appearing on many multiple sites may only rank well for the originator or most-authoritative site, according to how their algorithms identify authoritativeness.
• Pages which are hiding keywords or using misleading TITLE text or META descriptions may expect to be negatively rated. (I recently blogged in detail about the new emphasis on META descriptions.)
• It’s quite possible that Google could be using a process like TrustRank wherein their evaluators may rate a sample set of pages from a large site, and then they could use the resulting average trust rating across all the site’s pages. (Various of us have fantasized that the mysterious “IndyRank” element that was accidentally exposed in a Google error page could be referring to a human rating value for a site or page.) If this is occuring, your entire site could suffer if you have one section that has low quality according to their scoring guidelines.

The end result of these trends is that the practice of SEO is transforming from a stew of technical wizardry into applying methods for good user-centered design. Sure, we’ll still need to insure that pages will include actual text content, and expose database content through crawlable links. But tech tricks for increasing the relevancy of a page for a keyword may not be sustainable over the longterm.

Stephan Spencer and others have said that the SEO industry might not even survive long-term, because SEO is parasitically dependent upon the failures of designers to build usable websites, and the failures of search engines to perfectly rank pages for their subject matter.

I’ve attended Search Engine Strategies conferences for many years, and I’ve heard engineers from each of the major engines recommend that webmasters concentrate on usability more than on how to game the SERPs, but Google appears to be actually quantifying usability. If they master this, it really could be the end of the SEO industry. The other SEs will just play follow-the-leader when this happens.

Are you a SEO professional who is ready for this sea change? Are you educated about usability testing and user-centered design? One book I recommend on the subject is Steve Krug’s “Don’t Make Me Think”.

Initially, I thought that there wasn’t significant advantage to me for using these systems, because I’d already developed good methods for providing our page links to the search engines through the natural linking found in our site navigation systems.

Why should I expend yet more time and resources to dynamically produce the link files?

I have begun using these tools, though, because there are additional features now beyond just the URL disclosure pieces. Google’s Webmaster Tools include some nice reports on errors found when indexing, top keyword reports for sites, and page content analysis. My in-house analytics systems also have a lot of this same sort of reporting of course, but I’m interested in seeing Google’s perspective on my content.

As the SEs add even more webmaster tools, it’s eventually going to become necessary to fully integrate with them. It could easily come to the point where large sites will have to explicitly declare all the pages they wish to have indexed, just to insure that those pages might get ranked as optimally as possible.

Hopefully each of the major search engines will try to employ identical or compatible formats for site URLs, because it will be a hassle to have to keep up with multiple formats. This is an area where the SEs really ought to cooperate with one another for “pro bono publico” – for the common good. Currently, Yahoo seems to be just defensively immitating Google in this arena, and no one’s showing signs of collaborating. (At the recent SES Conference in San Jose, an audience member had a question for Yahoo’s Dr. Rajat Mukherjee, but the audience member kept referring to Yahoo’s product as “Yahoo Sitemaps” instead of Site Explorer, much to the consternation of Mukherjee. Amanda and others from Google who were sitting in front of me were highly amused at the situation. It was very obvious that the parallel Google and Yahoo teams have a healthy competitive streak betwixt them.)

If you’re reading this, perhaps you’re trying to decide if using Google’s Webmaster Tools or Yahoo Site Explorer will be valuable to you or not. Here’s my advice:

Register with these services so that you can use the tools and reports they offer. If you have a site that’s not already optimized and well-indexed, use the tools to provide them with all your page URLs. While use of their services isn’t a guarantee that your pages will be ranked well in the SERPs (“search engine results pages”), it’s a sure bet that if they can’t find your pages you won’t be in the SERPs at all. This can help you make sure your pages can get indexed. This is particularly useful if you are trying to get a brand new site indexed.

Also, keep up with the developments at the Google and Yahoo teams, because they’re each bound to deploy more tools and features as time progresses.

Google Webmaster Central blog

Yahoo! Search Blog

(MSN doesn’t have a webmaster tools portal yet, though they surely will at some point.)

If you’re one of the development team members from the search engines, I have to tell you what the next killer app tool could be: provide an interface that would allow us to see how our pages rank for at least 300 keywords. I know, I know — you guys don’t like providing a lot of metrics, since people will use it to test your black-box algorithms and figure out what/how various signals are used. The thing is, some people use 3rd-party software to accomplish this already (which you don’t like, since automated queries can impact performance), or some of us can hire temps to execute the searches and document the rankings. So, we’re getting this data already — you might as well provide it as a useful service to us, and to obviate the need for people or scripts to execute pointless searches.

I’m still very excited about the increasing functionality provided by the search engines, and I hope the trend continues.

As you may recall, I previously blogged a bit about the rise of geotagging, particularly geotagging of photos, and I had said that it seemed to be a really strong idea with a lot of potential uses. It’s gratifying to see that a service like flickr (and a company like Yahoo!) also believes that it will be strategically beneficial.

The number of people who have been geotagging or who even know about it is likely a relatively low percentage of the online populace, I’d guess (partly for the reason that most people don’t have a GPS device to tell them a location’s longitude and latitude). Now that a top-ranked photo site is supporting it expressly, droves of users will become educated about it, and experiment with it. By doing this, flickr is propelling the trend into the mainstream, increasing the likelihood that it’ll be more widely adopted.

Flickr’s new geotagging utilities were built by mashing-up their image management utilities with Yahoo! Maps, allowing users to drag pix onto a mapped location of where the image was taken in order to associate the photo with the geotag. Also, it appears that users could now use a graphic map as a navigational interface to browse geographic locations and then pull up any publicly-available photos associated with that location. Read on for more info.

To use the service, you go under their Organizr and click on the Map tab. When I first went in, I was pleased to see a message appear that stated that they noticed I’d already geotagged some photos, and would I like those to automatically be added to the map. Beautiful! Acknowledging the existing geotagged photos and allowing the user to choose to associate them with the map is top-notch usability. Also, the interface allows the photo owner to decide the photo’s level of privacy — so, top marks on the photo privacy as well! No one should want photos of their home (or, photos of their gold coin collection, for instance) to be associated with a map marking their location, and have that info to be publicly available!

Adding photos to the map was pretty easy — just pull up a map of the location where you shot a photo, select one of your pix from the image catalog at the bottom of the page, then drag the photo over the map to pinpoint it.

Flickr hides the newly-added geotags in the code of the image’s main page like this:

<span class=”geo” style=”display: none”/>
<span class=”latitude”>37.334089</span>,
<span class=”longitude”>-121.89051</span>

Those span tags are part of a map link that they add to the “Additional Information” segment of the page — allowing a small thumbnail map to pop up showing the location pinpoint. In flickr’s blog they mention that it will no longer be necessary to add the long/lat to the Tags of the image:

As a bonus there will be no more need for the unsightly “geotagged/geo:lat/geo:long” tags cluttering up your photos – we’ll offer an automated way to remove them all once the development community has had a chance to make the necessary changes to their code.

I have to express a bit of mild criticism about this decision, I guess. I think there are compelling reasons to visibly display the geotags in the text of the page for various uses. There are offline appliances that people might wish to type those codes into, for instance, and hiding them in the code of the page seems to be a bad idea to me. Just display the longitude and latitude in that section of “Additional Information”. A user might also desire to copy the location codes into another online application — perhaps they prefer using Mapquest’s driving directions, for instance.

Another negative note: like so many newly-launched cool apps, flickr/Yahoo are obviously unready for the influx of traffic hitting the new system. Maps are pulling up excruciatingly slowly, and the AJAX tiling of the map images (to allow the drag-panning features) is funky for the moment with only some of the tiles pulling up for any given map. Hopefully they’ll resolve this quickly, though.

One recommendation for the Flickr team: I’d like to be able to send other people the precise URL of the map navigation location that I browse to. The map navigation utility is great, but I can’t send folx a URL to the section of San Jose where I photographed and geotagged a picture of a church near the McEnery Convention Center, for instance.

For everyone else: I highly recommend trying out the new geotagging and browsing features at flickr! Local Search and Image Search have all converged with this evolutionary step, and we can expect to see new and interesting developments catalyzed by it.

If you’re curious, you can check out my geotagged pix on my personal map at flickr.

Predictibly, a number of folx have complained in the comments added to Yahoo’s “Weather Report” blog about the update. Jeremy Zawodny subsequently posted that their search team was paying close attention to the comments, which is always nice to hear.

Coincidentally, I’d also just recently posted about Google’s apparent use of page text to help identify a site’s overall authoritativeness for particular keywords/themes.

As they say, there’s nothing really new under the sun. I wonder if the search engines are all returning to the trend of authority/hub focus in algorithm development? It’s a strong concept and useful for ranking results, so the methodology for identifying authorities and hubs is likely here to stay.