• On TV.com: THE GIRLS NEXT DOOR photos

March 2, 2004 5:33 AM PST

Yahoo crawls deep into the Web

  • Print
Yahoo on Tuesday began a systematic effort to draw more content into its searchable database of Web documents, its latest bid to win Web surfers from search rival Google.

The Web portal, based in Sunnyvale, Calif., introduced its Content Acquisition Program designed to index the billions of documents contained in public databases


News.Commentary
Google's soft spot
There are limits to Google's
ambitions--and ways for portals
to gain an advantage.

but that are commonly inaccessible to search engines, or what's called the invisible or deep Web. To this end, it has aligned with the Library of Congress, the University of California at Los Angeles, National Public Radio, the University of Michigan and Project Gutenberg, among others, to begin seeding its index with fresh, searchable material for Web surfers' queries.

"We aim to extend the way we discover content for free," said Tim Cadogan, Yahoo's vice president of search. "One of the challenges is that the interaction between content providers and search engines is lacking. So we said, let's reach out to the public domains and nonprofits and try to get more of that content exposed."

There's also a commercial component to the new program, called Site Match. For a fee, Yahoo will let marketers pay to regularly feed it Web addresses for speedier indexing in the database, a practice in the industry called paid inclusion. Yahoo has essentially repackaged the paid-inclusion programs of its three recent acquisitions--Inktomi, AltaVista and Fast Web search--to produce Site Match. Marketers will now be able to buy into one indexing program, instead of three, to be included in Yahoo search results as well as those of search partners like MSN.

The move comes only weeks after Yahoo dumped Google as its search provider and began using its own technology platform. Yahoo had been gearing up for the switch for the past year, after buying search technology company Inktomi and commercial listings provider Overture Services for nearly $2 billion in a bid to recapture the search crown from Google.

The company has a vested interest in winning back the loyalty of Web searchers it lost to Google in recent years; financial analysts expect that search engine advertising will generate nearly $4 billion this year.

The move also comes weeks after Google boasted the expansion of its own searchable index to more than 6 billion documents. Google said at the time that growing its database serves to improve the overall comprehensiveness of its search results.

Yahoo's Cadogan would not say how many documents it searches--only that its index reaches into several billion. He said that about 99 percent of the company's search results draw on documents it has obtained freely from public Web sites. The other 1 percent is the result of the paid-inclusion program.

The new content program will help Yahoo increase the comprehensiveness of the index by tapping the estimated 10 billion to 100 billion documents of the deep Web, he said. In one partnership, Yahoo will receive feeds of more than 17,000 pieces of related audio content from NPR, the nationally syndicated news radio program. It will also receive fresh feeds from NPR so that it can index new content within one to two days.

Cadogan said that Web sites like eBay can also pay to participate in Site Match to ensure that an auction listing is current when a Web surfer searches on "BMW 2002 green," for example.

Site Match costs marketers $49 annually to list the first Web address and $29 for the second Web address thereafter, up to 10. After 10 addresses, the site would pay $10 per listing. The marketer then pays between 15 cents and 30 cents when a Web surfer clicks on its listing in search results.

Still, some industry watchers have expressed concern that paid-inclusion programs can adversely influence search results and confuse Web surfers who are looking for unbiased results.

Google does not offer a paid-inclusion program and its executives have denounced for-fee indexing because of its potential to skew results.

Cadogan maintained that there is an "iron wall" between commercial and free search results on Yahoo, governed by quality standards and ranking algorithms that ensure that results are relevant to people.

"Everything goes through a quality process, including an example in the free crawl," he said, "and every piece of content is handled exactly the same way for ranking."

See more CNET content tagged:
Inktomi, Yahoo! Inc., index, bid, search engine company

Add a Comment (Log in or register) 5 comments
Is there a sanbox effect for all search engines
by February 9, 2005 4:45 AM PST
Is there a sanbox effect for all search engines
I have a site that's has many links from high PR sites and a great link on the homepage of a site that is in the top 5 for my key phrase on Google, MSN and Yahoo but my site ranks no better than 500 plus. Is this because my site is new?
Reply to this comment
links from high PR sites
by Rebeca Stamos May 28, 2007 3:47 PM PDT
http://www.analogstereo.com/nissan_owners_manual.htm
View reply
I believe there is.
by April 24, 2005 8:29 AM PDT
It seems simple to me now, there is. Cas in point http://www.keysheet.com for paid surveys
Reply to this comment
Firefox and Internet Explorer
by jordan357 June 16, 2007 10:52 AM PDT
http://www.highspeedsat.com/exploristxl.htm
Magellan - explorist XL satellite handheld GPS unit is powerful, affordable GPS with track-plotter. Explorist-xl provides reliable accuracy to within 3 meters. Save track logs to guide you to your favorite spots. Back Track will lead you to camp, the car and home.
Reply to this comment
advertisement

Latest tech news headlines

RSS Feeds

Add headlines from CNET News to your homepage or feedreader.

More feeds available in our RSS feed index.

Markets

Market news, charts, SEC filings, and more

Related quotes

Google (1.11%) 2.87 262.43
Yahoo (4.92%) 0.44 9.39
Dow Jones Industrials (6.54%) 494.13 8,046.42
S&P 500 (6.32%) 47.59 800.03
NASDAQ (5.18%) 68.23 1,384.35
CNET TECH (5.95%) 56.25 1,002.00
  Symbol Lookup
advertisement

Inside CNET News

Scroll Left Scroll Right