A Short History of Online Search
Like this article? We recommend
Like this article? We recommend
Like this article? We recommend
When most people think online search, they think Google. That's what's happened to the online search market – it's dominated by one (very good) player, with a few minor competitors (Bing and Yahoo!) picking up the scraps.
But it wasn't always this way. There was a time when Google was just an interesting experiment, and users had a dozen or more viable search engines and directories to choose from.
So strap on your goggles (not your Googles!) and get ready to take a trip into the search engine past – way back into the mid-1990s!
Searching the Internet – Before the Web
The Internet today is primarily the World Wide Web – that "www" in all website addresses. But the Internet was around long before web pages and hyperlinks became common, since the 1970s in one form or another.
Users back in those pre-Web days still had the same issues that users today face – namely, sorting through all the junk to find the golden nuggets of information they needed. Just as now, information back then was stored on a multitude of servers around the globe, and being able to search those servers for specific information was an essential task.
If you've never seen the pre-Web Internet, know that it wasn't pretty. There weren't any fancy graphics and clickable hyperlinks, which means that all the data back then existed in text format only. And there weren't any search engines per se, not like Google or Bing, anyway.
So how did users search for information before there was the Web? They used one of four primary tools:
- WAIS, which stands for wide area information server. This tool let you use the old text-based Telnet protocol to perform full-text document searches of various Internet servers.
- Archie, which was a client for searching for files across multiple FTP sites. (The word "archie" is the word "archive" with the "v" removed.) FTP sites still exist, but Archie is long gone.
- Gopher, which was a tool for organizing files on dedicated servers. Gopher was surprisingly popular in universities across the U.S., which is where most of the information back then was housed. (Gopher was created at the University of Minnesota – home of the Golden Gophers.) Each Gopher server contained lists of files and other information, both at that specific site and at other Gopher sites around the world. Gopher worked similarly to a hierarchical file tree like that used in Windows Explorer – you clicked folder links to see their contents and navigated up and down through various folders and subfolders.
- Veronica. When you wanted to find information on a specific Gopher server, you used a Gopher client for that server. But when you wanted to search across multiple Gopher servers, you used Veronica. Veronica was the Archie for Gopher servers. (Archie and Veronica – get it?) This software client functioned kind of like one of today's search engines – you entered a query and clicked a Search button, which generated a list of matching documents found on various Gopher servers.
)
Figure 1 Searching the servers at the University of Illinois (Urbana-Champaign) with a Gopher client
These tools were all rather primitive, at least by today's standards. And after the Web came along, these old tools went the way of the horse carriage and buggy whip.
Enter the Web – and Web Directories
With the advent of the World Wide Web in 1994 (or thereabouts), data started migrating from Gopher and FTP servers to Web servers. Boring old text documents got dusted off and spruced up with graphics and hyperlinks, and Microsoft and Netscape started battling back and forth about who had the better Web browser. (This was after Mosaic paved the way, of course.) In short, the Internet was stood on its head as the Web became the dominant infrastructure – and as millions of new users flooded the Internet monthly.
As the number of individual Web pages grew from tens of thousands to hundreds of thousands to millions, it became imperative for people to quickly and easily find their way around all those pages. With the explosion of the Web, then, came a new industry of cataloging and indexing the Web.
The earliest attempts to catalog the Web were all done manually. That's much different from the way today's search engines do it, with automated web crawlers and indexing software. Instead, back in the day, real honest-to-goodness human beings physically looked at individual websites and pages and manually stuck each one into a hand-picked category. When they got enough Web pages collected, they ended up with what was called a directory.
It's important to know that a directory (and there are still a few around today) doesn't search the Web, it only catalogs chosen Web pages, which themselves represent a small subset of everything available. But a directory is very organized and very easy to use, and lots and lots of people back in the mid-90s used Web directories every day.
In many ways, those Web directories looked and worked like traditional print Yellow Pages. (Which are also facing extinction today, by the way.) When you wanted to find something, you clicked through the various categories and subcategories on a given directory site until you ended up with a list of pages recommended by the directory's editors. You didn't get the magnitude of results you get today, but what you got were choice. It's the old quality versus quantity thing.
One of the first Web directories, launched late in 1993, was called ALIWEB. (That that stands for Archie-Like Indexing for the WEB.) Where later directories used editors to hand pick Web pages, ALIWEB assembled its index from user submissions. As relatively few people submitted pages to the directory, ALIWEB was not widely used and had a short existence.
)
Figure 2 ALIWEB's home page, circa 1993
Other early directories included LookSmart, EINet Galaxy, and Best of the Web – the latter the Web's first directory consisting solely of paid submissions. All of these either evolved into search engines, got acquired, or went out of business – all within the course of a half-dozen years.
The biggest directory of them all, however, was Yahoo!, launched in 1994. That was the year that David Filo and Jerry Yang, two grad students at Stanford University, first assembled their own directory of Web pages, originally dubbed "David and Jerry's Guide to the World Wide Web." (They changed the name to the more-recognizable Yahoo! a year later.)
)
Figure 3 An early version of the Yahoo! Directory, circa 1996
What Yahoo! had going for it was organization – a hierarchical organization. So, while you could search the directory, you could also browse it by clicking through the multiple levels of a category's hierarchy. That functionality, along with the biggest index of pages of any other directory of the time, fueled Yahoo's rapid growth. By 1998, Yahoo! (which had added a portal-type home page interface) had become the number-one gateway page for web users of the time.
Yahoo! didn't exclusively remain a directory for long, however. In addition to the portal-like interface and other services launched earlier, in the year 2000 Yahoo! added search engine results to its directory listings. (Interestingly, those search results were provided by new competitor Google.) The Yahoo! Directory exists to this day (at dir.yahoo.com), although it's a very minor part of the current Yahoo! empire.
Another directory that survived the transition to search engines is the Open Directory Project, also known as DMOZ (from the Web address directory.mozilla.org). Back in 1998, Rich Skenta and Bob Truel, two engineers at Sun Microsystems, grew frustrated with the lengthy times it took Yahoo! to add new pages to its Web directory. Thus the Open Directory was born, using volunteer editors to assemble its database of entries.
Netscape acquired the Open Directory Project later in November, 1998; Netscape was itself acquired by AOL just a month later. For a time, the Open Directory Project was licensed by other major websites (including Google, for its not-well-known Google Directory), although it's pretty much a freestanding directory these days. The Open Directory remains open, still maintained by a core group of loyal volunteers. Check it out yourself at www.dmoz.org.
)
Figure 4 The Open Directory today
Search Engines – a Better Way to Search the Web
Directories made sense when the Web itself was still relatively small and manageable. But as the number of websites and pages multiplied, there was no way a human-edited directory could ever keep pace. (Yahoo!, arguably the biggest directory of its day, cataloged fewer than 1/100th of 1% of the total number of Web pages then published.)
There had to be a more effective and efficient way to search all those hundreds of millions of Web pages – and there was. It was called a search engine.
Unlike manually edited Web directories, a search engineis not powered by human hands; instead, a search engine uses a special type of software program (called a spider or a crawler) to roam the Web automatically, feeding what it finds back to a massive bank of computers. These computers create a huge index (actually, a database) of what the robots find, in the form of keywords and abridged content from each page.
When a user enters a query into one of these search engines, that query is matched against the index of pages held on the search engine's servers. (The search engine doesn't actually search the entire web every time a query is entered.) Those pages from the index that best match the query are displayed on the search results page; clicking one of the results opens the original page on the Web.
The first Web search engine, dubbed WebCrawler, was launched in 1994. Developed by Brian Pinkerton of the University of Washington, WebCrawler became so popular that you couldn't actually access during daytime hours. WebCrawler was acquired by America Online in 1995, and used to power AOL Search for several years. AOL sold WebCrawler to Excite two years later, but by that time there was plenty of competition in the Web search market.
)
Figure 5 WebCrawler, the first search engine, in 1994
Lycos was one of those early competitors. Also launched in 1994, Lycos was developed by Michael Maudlin of Carnegie Mellon University. Lycos got its edge from the size of its index, which by November, 1996, had grown to 60 million pages, far more than any other competing search engine.
)
Figure 6 The Lycos website in 1996
Infoseek was another search engine launched in 1994, by entrepreneur Steve Kirsch. While Infoseek didn't bring a lot of technical innovation to the game, the company was aggressive in seeking out partners – chief of which was Netscape, which featured Infoseek as the default search engine for its self-named Web browser.
A more important player launched the same month as Infoseek. AltaVista was developed by engineers at Digital Equipment Corporation, and brought multiple innovations to the search engine market. These included natural language queries, advanced searching techniques, and inbound link checking. Then there was the near-unlimited bandwidth provided by DEC, which resulted in faster performance without pesky slowdowns during high traffic periods.
)
Figure 7 The AltaVista search engine, circa 1996
All of these things gave AltaVista a huge competitive advantage, and made it the search engine of choice for a period. That's until questionable management, interface clutter, and other missteps drove customers to competing search engines. By 2003, AltaVista was a relic, purchased by Yahoo! which pretty much dismantled the technology behind the search engine.
HotBot was the search engine that displaced AltaVista at the top of the heap. This fast new search engine was launched in 1996 by Inktomi Corporation, based on a school project of UC Berkeley professor Eric Brewer and grad student Paul Gauthier. HotBot was hot stuff for a period, and Inktomi did a good job licensing its search technology to other sites – and developed a successful search advertising business, to boot.
)
Figure 8 HotBot in 1997
Inktomi/HotBot remained the most popular search engine until the rise of Google a few years later. The company's decline was exacerbated by the bursting of the dot-com bubble at the end of the decade, which contributed to its acquisition by Yahoo! in 2003.
Other interesting search engines gained popularity during this period. Excite was actually launched, by six Stanford undergrads, back in 1993, but only came to prominence after purchasing two existing search engines (WebCrawler and Magellan) in 1996. The company's big coup was signing distribution agreements with Netscape, Microsoft, and other players, which let it play in the big leagues for a while. Excite was purchased by broadband provider @Home in 1999, filed for bankruptcy in 2001, and was picked up by Infospace (for pennies on the dollar) later that year.
)
Figure 9 The Excite website in 1998
I personally was a fan of the Northern Light search engine, which was also launched in 1996. Northern Light was innovative in the way it classified its search results, and the fact that it included information from both public and proprietary sources. That made it good for business-oriented searches, or so I found. Unfortunately, Northern Light never gained much traction among consumers, and in 2002 the company discontinued the public search engine in favor of an enterprise search product.
)
Figure 10 The Northern Light search engine in 1997
Ask Jeeves (known later as Ask.com and now just Ask) was launched in 1997 and gained recognition for its natural language search engine. Searches were initially powered by technology licensed from DirectHit, which tried to rank results based on popularity. In 2000, Ask Jeeves replaced DirectHit with Teoma search technology, which used clustering to organize search results by subject popularity. The results improved a little, but not enough to make a dent in the market at the time. In 2005, the company was acquired by Barry Diller's Interactive Search Holding company and the Ask search site continues to this day as part of the IAC portfolio, alongside other sites such as Match.com and Ticketmaster.com.
)
Figure 11 Ask Jeeves, circa 1999
About the same time that Ask Jeeves came to prominence, Norwegian student Tor Egge developed an enterprise search engine known as FAST (for Fast Search and Transfer). FAST technology was used to power the AllTheWeb site, which launched in 1999. FAST/AllTheWeb produced quality results fast and had a brief fling with users' attentions, but quickly faded and was acquired by Overture in 2003. When Yahoo! later acquired Overture they rolled out some of the FAST technology into Yahoo! Search.
Then there was Microsoft, which has always lusted after online riches. (Windows and Office just weren't enough for the Microsofties, were they?) Microsoft launched MSN Search in 1998, which incorporated search results from Overture, LookSmart, and Inktomi. It was not an immediate success. That didn't deter Microsoft, however, which replaced MSN Search with Live Search in 2006, and then with Bing in 2009. Bing continues as a distant number-two player today.
)
Figure 12 MSN Search in 1999
)
Figure 13 Bing search today
Interestingly, if you submitted the exact same query to several of these early search engines, they'd often return completely different results. That's because each search engine had its method of crawling the web and its own unique approach to ranking thee results. This led to the creation of metasearch engines, such as Dogpile and MetaCrawler. These sites let you submit a single query to multiple search engines, and then see the results from each on a single page. Not very elegant, but often necessary back in the day.
)
Figure 14 MetaCrawler, circa 2000, searching the search engines
Google Conquers the World
While search engine companies sprang up like wild flowers in the 1990s, few stood the test of time. Most folded or got acquired, and those handful that remain today failed to grab a long-term leadership position in the online search market. That position was claimed by one of the later entrants, a little company that called itself Google.
In January 1996, Larry Page and Sergei Brin, two Ph.D. students at Stanford University, started working on a search engine that utilized links from other websites (called backlinks) to better rank search results. It's a kind of popularity contest; the more sites that link to a given page, the more relevant that page must be, and thus the higher it ranks in search results.
Page and Brin dubbed their project BackRub, and it quickly gained a reputation for providing more reliable and targeted search results than other search engines of the time. In 1998 Page and Brin renamed BackRub as Google, and officially launched it for public use.
The company tried to license its search ranking technology (called PageRank) to other search sites, but had no takers. This led the company to more strongly push its own Google search site, and to find a way to make money from its search services – which it did, in the form of search advertising. But that's another story.
)
Figure 15 An early Google home page, circa 1999
What Google was effective at was building its own customer base while at the same time partnering with other sites to provide better-quality search results. For example, Google became AOL's search partner in 1999, and Yahoo's in 2000.
The result is that Google slowly but surely vanquished all major competitors. The AltaVistas and the HotBots and the Excites of the world have all faded away, leaving online search in the hands of Google and a handful of minor competitors (from major companies).
Looking at the numbers, Google owns about two-thirds of the current market for online search, which makes it somewhat ubiquitous. (We don't search for things anymore, we "Google" them.) Bing is the number-two player with about 18 percent share, while Yahoo! has shrunk to a poor number three, with just 11 percent of the market. All the other search engines – including Ask and AOL Search – have low single-digit shares.
)
Figure 16 Google today – just as minimalist as ever
The online search market has long moved beyond evaluating success based on the raw number of index entries; it's now all about the relevance of search results, in which Google continues to lead the pack. Google has continued to evolve its search technology to provide ever-more-targeted results, adapting to the introduction of social media and an increasingly mobile user base.
So even though Google has expanded far beyond online search – we're talking cloud-based office suites, mapping and photo editing software, and even self-driving cars – the company remains known for its search engine, and rightly so. It's been an interesting two decades of online search, but – thanks to Google and other companies through the years – it's now easier than ever before to get your questions answered and find information out there on the World Wide Web.