Home > Articles

A Short History of Online Search

Remember AltaVista? HotBot? Lycos? When most people think online search, they think Google. That's what's happened to the online search market — it's dominated by one (very good) player, with a few minor competitors (Bing and Yahoo!) picking up the scraps. In this article, old-timer Michael Miller takes you back in time to revisit the short but significant history of online search engines, from Archie to Yahoo! — and everything in-between.
Like this article? We recommend

When most people think online search, they think Google. That's what's happened to the online search market – it's dominated by one (very good) player, with a few minor competitors (Bing and Yahoo!) picking up the scraps.

But it wasn't always this way. There was a time when Google was just an interesting experiment, and users had a dozen or more viable search engines and directories to choose from.

So strap on your goggles (not your Googles!) and get ready to take a trip into the search engine past – way back into the mid-1990s!

Searching the Internet – Before the Web

The Internet today is primarily the World Wide Web – that "www" in all website addresses. But the Internet was around long before web pages and hyperlinks became common, since the 1970s in one form or another.

Users back in those pre-Web days still had the same issues that users today face – namely, sorting through all the junk to find the golden nuggets of information they needed. Just as now, information back then was stored on a multitude of servers around the globe, and being able to search those servers for specific information was an essential task.

If you've never seen the pre-Web Internet, know that it wasn't pretty. There weren't any fancy graphics and clickable hyperlinks, which means that all the data back then existed in text format only. And there weren't any search engines per se, not like Google or Bing, anyway.

So how did users search for information before there was the Web? They used one of four primary tools:

  • WAIS, which stands for wide area information server. This tool let you use the old text-based Telnet protocol to perform full-text document searches of various Internet servers.
  • Archie, which was a client for searching for files across multiple FTP sites. (The word "archie" is the word "archive" with the "v" removed.) FTP sites still exist, but Archie is long gone.
  • Gopher, which was a tool for organizing files on dedicated servers. Gopher was surprisingly popular in universities across the U.S., which is where most of the information back then was housed. (Gopher was created at the University of Minnesota – home of the Golden Gophers.) Each Gopher server contained lists of files and other information, both at that specific site and at other Gopher sites around the world. Gopher worked similarly to a hierarchical file tree like that used in Windows Explorer – you clicked folder links to see their contents and navigated up and down through various folders and subfolders.
  • Figure 1 Searching the servers at the University of Illinois (Urbana-Champaign) with a Gopher client

  • Veronica. When you wanted to find information on a specific Gopher server, you used a Gopher client for that server. But when you wanted to search across multiple Gopher servers, you used Veronica. Veronica was the Archie for Gopher servers. (Archie and Veronica – get it?) This software client functioned kind of like one of today's search engines – you entered a query and clicked a Search button, which generated a list of matching documents found on various Gopher servers.

These tools were all rather primitive, at least by today's standards. And after the Web came along, these old tools went the way of the horse carriage and buggy whip.

Enter the Web – and Web Directories

With the advent of the World Wide Web in 1994 (or thereabouts), data started migrating from Gopher and FTP servers to Web servers. Boring old text documents got dusted off and spruced up with graphics and hyperlinks, and Microsoft and Netscape started battling back and forth about who had the better Web browser. (This was after Mosaic paved the way, of course.) In short, the Internet was stood on its head as the Web became the dominant infrastructure – and as millions of new users flooded the Internet monthly.

As the number of individual Web pages grew from tens of thousands to hundreds of thousands to millions, it became imperative for people to quickly and easily find their way around all those pages. With the explosion of the Web, then, came a new industry of cataloging and indexing the Web.

The earliest attempts to catalog the Web were all done manually. That's much different from the way today's search engines do it, with automated web crawlers and indexing software. Instead, back in the day, real honest-to-goodness human beings physically looked at individual websites and pages and manually stuck each one into a hand-picked category. When they got enough Web pages collected, they ended up with what was called a directory.

It's important to know that a directory (and there are still a few around today) doesn't search the Web, it only catalogs chosen Web pages, which themselves represent a small subset of everything available. But a directory is very organized and very easy to use, and lots and lots of people back in the mid-90s used Web directories every day.

In many ways, those Web directories looked and worked like traditional print Yellow Pages. (Which are also facing extinction today, by the way.) When you wanted to find something, you clicked through the various categories and subcategories on a given directory site until you ended up with a list of pages recommended by the directory's editors. You didn't get the magnitude of results you get today, but what you got were choice. It's the old quality versus quantity thing.

One of the first Web directories, launched late in 1993, was called ALIWEB. (That that stands for Archie-Like Indexing for the WEB.) Where later directories used editors to hand pick Web pages, ALIWEB assembled its index from user submissions. As relatively few people submitted pages to the directory, ALIWEB was not widely used and had a short existence.

Figure 2 ALIWEB's home page, circa 1993

Other early directories included LookSmart, EINet Galaxy, and Best of the Web – the latter the Web's first directory consisting solely of paid submissions. All of these either evolved into search engines, got acquired, or went out of business – all within the course of a half-dozen years.

The biggest directory of them all, however, was Yahoo!, launched in 1994. That was the year that David Filo and Jerry Yang, two grad students at Stanford University, first assembled their own directory of Web pages, originally dubbed "David and Jerry's Guide to the World Wide Web." (They changed the name to the more-recognizable Yahoo! a year later.)

Figure 3 An early version of the Yahoo! Directory, circa 1996

What Yahoo! had going for it was organization – a hierarchical organization. So, while you could search the directory, you could also browse it by clicking through the multiple levels of a category's hierarchy. That functionality, along with the biggest index of pages of any other directory of the time, fueled Yahoo's rapid growth. By 1998, Yahoo! (which had added a portal-type home page interface) had become the number-one gateway page for web users of the time.

Yahoo! didn't exclusively remain a directory for long, however. In addition to the portal-like interface and other services launched earlier, in the year 2000 Yahoo! added search engine results to its directory listings. (Interestingly, those search results were provided by new competitor Google.) The Yahoo! Directory exists to this day (at dir.yahoo.com), although it's a very minor part of the current Yahoo! empire.

Another directory that survived the transition to search engines is the Open Directory Project, also known as DMOZ (from the Web address directory.mozilla.org). Back in 1998, Rich Skenta and Bob Truel, two engineers at Sun Microsystems, grew frustrated with the lengthy times it took Yahoo! to add new pages to its Web directory. Thus the Open Directory was born, using volunteer editors to assemble its database of entries.

Netscape acquired the Open Directory Project later in November, 1998; Netscape was itself acquired by AOL just a month later. For a time, the Open Directory Project was licensed by other major websites (including Google, for its not-well-known Google Directory), although it's pretty much a freestanding directory these days. The Open Directory remains open, still maintained by a core group of loyal volunteers. Check it out yourself at www.dmoz.org.

Figure 4 The Open Directory today

Search Engines – a Better Way to Search the Web

Directories made sense when the Web itself was still relatively small and manageable. But as the number of websites and pages multiplied, there was no way a human-edited directory could ever keep pace. (Yahoo!, arguably the biggest directory of its day, cataloged fewer than 1/100th of 1% of the total number of Web pages then published.)

There had to be a more effective and efficient way to search all those hundreds of millions of Web pages – and there was. It was called a search engine.

Unlike manually edited Web directories, a search engineis not powered by human hands; instead, a search engine uses a special type of software program (called a spider or a crawler) to roam the Web automatically, feeding what it finds back to a massive bank of computers. These computers create a huge index (actually, a database) of what the robots find, in the form of keywords and abridged content from each page.

When a user enters a query into one of these search engines, that query is matched against the index of pages held on the search engine's servers. (The search engine doesn't actually search the entire web every time a query is entered.) Those pages from the index that best match the query are displayed on the search results page; clicking one of the results opens the original page on the Web.

The first Web search engine, dubbed WebCrawler, was launched in 1994. Developed by Brian Pinkerton of the University of Washington, WebCrawler became so popular that you couldn't actually access during daytime hours. WebCrawler was acquired by America Online in 1995, and used to power AOL Search for several years. AOL sold WebCrawler to Excite two years later, but by that time there was plenty of competition in the Web search market.

Figure 5 WebCrawler, the first search engine, in 1994

Lycos was one of those early competitors. Also launched in 1994, Lycos was developed by Michael Maudlin of Carnegie Mellon University. Lycos got its edge from the size of its index, which by November, 1996, had grown to 60 million pages, far more than any other competing search engine.

Figure 6 The Lycos website in 1996

Infoseek was another search engine launched in 1994, by entrepreneur Steve Kirsch. While Infoseek didn't bring a lot of technical innovation to the game, the company was aggressive in seeking out partners – chief of which was Netscape, which featured Infoseek as the default search engine for its self-named Web browser.

A more important player launched the same month as Infoseek. AltaVista was developed by engineers at Digital Equipment Corporation, and brought multiple innovations to the search engine market. These included natural language queries, advanced searching techniques, and inbound link checking. Then there was the near-unlimited bandwidth provided by DEC, which resulted in faster performance without pesky slowdowns during high traffic periods.

Figure 7 The AltaVista search engine, circa 1996

All of these things gave AltaVista a huge competitive advantage, and made it the search engine of choice for a period. That's until questionable management, interface clutter, and other missteps drove customers to competing search engines. By 2003, AltaVista was a relic, purchased by Yahoo! which pretty much dismantled the technology behind the search engine.

HotBot was the search engine that displaced AltaVista at the top of the heap. This fast new search engine was launched in 1996 by Inktomi Corporation, based on a school project of UC Berkeley professor Eric Brewer and grad student Paul Gauthier. HotBot was hot stuff for a period, and Inktomi did a good job licensing its search technology to other sites – and developed a successful search advertising business, to boot.

Figure 8 HotBot in 1997

Inktomi/HotBot remained the most popular search engine until the rise of Google a few years later. The company's decline was exacerbated by the bursting of the dot-com bubble at the end of the decade, which contributed to its acquisition by Yahoo! in 2003.

Other interesting search engines gained popularity during this period. Excite was actually launched, by six Stanford undergrads, back in 1993, but only came to prominence after purchasing two existing search engines (WebCrawler and Magellan) in 1996. The company's big coup was signing distribution agreements with Netscape, Microsoft, and other players, which let it play in the big leagues for a while. Excite was purchased by broadband provider @Home in 1999, filed for bankruptcy in 2001, and was picked up by Infospace (for pennies on the dollar) later that year.

Figure 9 The Excite website in 1998

I personally was a fan of the Northern Light search engine, which was also launched in 1996. Northern Light was innovative in the way it classified its search results, and the fact that it included information from both public and proprietary sources. That made it good for business-oriented searches, or so I found. Unfortunately, Northern Light never gained much traction among consumers, and in 2002 the company discontinued the public search engine in favor of an enterprise search product.

Figure 10 The Northern Light search engine in 1997

Ask Jeeves (known later as Ask.com and now just Ask) was launched in 1997 and gained recognition for its natural language search engine. Searches were initially powered by technology licensed from DirectHit, which tried to rank results based on popularity. In 2000, Ask Jeeves replaced DirectHit with Teoma search technology, which used clustering to organize search results by subject popularity. The results improved a little, but not enough to make a dent in the market at the time. In 2005, the company was acquired by Barry Diller's Interactive Search Holding company and the Ask search site continues to this day as part of the IAC portfolio, alongside other sites such as Match.com and Ticketmaster.com.

Figure 11 Ask Jeeves, circa 1999

About the same time that Ask Jeeves came to prominence, Norwegian student Tor Egge developed an enterprise search engine known as FAST (for Fast Search and Transfer). FAST technology was used to power the AllTheWeb site, which launched in 1999. FAST/AllTheWeb produced quality results fast and had a brief fling with users' attentions, but quickly faded and was acquired by Overture in 2003. When Yahoo! later acquired Overture they rolled out some of the FAST technology into Yahoo! Search.

Then there was Microsoft, which has always lusted after online riches. (Windows and Office just weren't enough for the Microsofties, were they?) Microsoft launched MSN Search in 1998, which incorporated search results from Overture, LookSmart, and Inktomi. It was not an immediate success. That didn't deter Microsoft, however, which replaced MSN Search with Live Search in 2006, and then with Bing in 2009. Bing continues as a distant number-two player today.

Figure 12 MSN Search in 1999

Figure 13 Bing search today

Interestingly, if you submitted the exact same query to several of these early search engines, they'd often return completely different results. That's because each search engine had its method of crawling the web and its own unique approach to ranking thee results. This led to the creation of metasearch engines, such as Dogpile and MetaCrawler. These sites let you submit a single query to multiple search engines, and then see the results from each on a single page. Not very elegant, but often necessary back in the day.

Figure 14 MetaCrawler, circa 2000, searching the search engines

Google Conquers the World

While search engine companies sprang up like wild flowers in the 1990s, few stood the test of time. Most folded or got acquired, and those handful that remain today failed to grab a long-term leadership position in the online search market. That position was claimed by one of the later entrants, a little company that called itself Google.

In January 1996, Larry Page and Sergei Brin, two Ph.D. students at Stanford University, started working on a search engine that utilized links from other websites (called backlinks) to better rank search results. It's a kind of popularity contest; the more sites that link to a given page, the more relevant that page must be, and thus the higher it ranks in search results.

Page and Brin dubbed their project BackRub, and it quickly gained a reputation for providing more reliable and targeted search results than other search engines of the time. In 1998 Page and Brin renamed BackRub as Google, and officially launched it for public use.

The company tried to license its search ranking technology (called PageRank) to other search sites, but had no takers. This led the company to more strongly push its own Google search site, and to find a way to make money from its search services – which it did, in the form of search advertising. But that's another story.

Figure 15 An early Google home page, circa 1999

What Google was effective at was building its own customer base while at the same time partnering with other sites to provide better-quality search results. For example, Google became AOL's search partner in 1999, and Yahoo's in 2000.

The result is that Google slowly but surely vanquished all major competitors. The AltaVistas and the HotBots and the Excites of the world have all faded away, leaving online search in the hands of Google and a handful of minor competitors (from major companies).

Looking at the numbers, Google owns about two-thirds of the current market for online search, which makes it somewhat ubiquitous. (We don't search for things anymore, we "Google" them.) Bing is the number-two player with about 18 percent share, while Yahoo! has shrunk to a poor number three, with just 11 percent of the market. All the other search engines – including Ask and AOL Search – have low single-digit shares.

Figure 16 Google today – just as minimalist as ever

The online search market has long moved beyond evaluating success based on the raw number of index entries; it's now all about the relevance of search results, in which Google continues to lead the pack. Google has continued to evolve its search technology to provide ever-more-targeted results, adapting to the introduction of social media and an increasingly mobile user base.

So even though Google has expanded far beyond online search – we're talking cloud-based office suites, mapping and photo editing software, and even self-driving cars – the company remains known for its search engine, and rightly so. It's been an interesting two decades of online search, but – thanks to Google and other companies through the years – it's now easier than ever before to get your questions answered and find information out there on the World Wide Web. 

InformIT Promotional Mailings & Special Offers

I would like to receive exclusive offers and hear about products from InformIT and its family of brands. I can unsubscribe at any time.


Pearson Education, Inc., 221 River Street, Hoboken, New Jersey 07030, (Pearson) presents this site to provide information about products and services that can be purchased through this site.

This privacy notice provides an overview of our commitment to privacy and describes how we collect, protect, use and share personal information collected through this site. Please note that other Pearson websites and online products and services have their own separate privacy policies.

Collection and Use of Information

To conduct business and deliver products and services, Pearson collects and uses personal information in several ways in connection with this site, including:

Questions and Inquiries

For inquiries and questions, we collect the inquiry or question, together with name, contact details (email address, phone number and mailing address) and any other additional information voluntarily submitted to us through a Contact Us form or an email. We use this information to address the inquiry and respond to the question.

Online Store

For orders and purchases placed through our online store on this site, we collect order details, name, institution name and address (if applicable), email address, phone number, shipping and billing addresses, credit/debit card information, shipping options and any instructions. We use this information to complete transactions, fulfill orders, communicate with individuals placing orders or visiting the online store, and for related purposes.


Pearson may offer opportunities to provide feedback or participate in surveys, including surveys evaluating Pearson products, services or sites. Participation is voluntary. Pearson collects information requested in the survey questions and uses the information to evaluate, support, maintain and improve products, services or sites, develop new products and services, conduct educational research and for other purposes specified in the survey.

Contests and Drawings

Occasionally, we may sponsor a contest or drawing. Participation is optional. Pearson collects name, contact information and other information specified on the entry form for the contest or drawing to conduct the contest or drawing. Pearson may collect additional personal information from the winners of a contest or drawing in order to award the prize and for tax reporting purposes, as required by law.


If you have elected to receive email newsletters or promotional mailings and special offers but want to unsubscribe, simply email information@informit.com.

Service Announcements

On rare occasions it is necessary to send out a strictly service related announcement. For instance, if our service is temporarily suspended for maintenance we might send users an email. Generally, users may not opt-out of these communications, though they can deactivate their account information. However, these communications are not promotional in nature.

Customer Service

We communicate with users on a regular basis to provide requested services and in regard to issues relating to their account we reply via email or phone in accordance with the users' wishes when a user submits their information through our Contact Us form.

Other Collection and Use of Information

Application and System Logs

Pearson automatically collects log data to help ensure the delivery, availability and security of this site. Log data may include technical information about how a user or visitor connected to this site, such as browser type, type of computer/device, operating system, internet service provider and IP address. We use this information for support purposes and to monitor the health of the site, identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents and appropriately scale computing resources.

Web Analytics

Pearson may use third party web trend analytical services, including Google Analytics, to collect visitor information, such as IP addresses, browser types, referring pages, pages visited and time spent on a particular site. While these analytical services collect and report information on an anonymous basis, they may use cookies to gather web trend information. The information gathered may enable Pearson (but not the third party web trend services) to link information with application and system log data. Pearson uses this information for system administration and to identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents, appropriately scale computing resources and otherwise support and deliver this site and its services.

Cookies and Related Technologies

This site uses cookies and similar technologies to personalize content, measure traffic patterns, control security, track use and access of information on this site, and provide interest-based messages and advertising. Users can manage and block the use of cookies through their browser. Disabling or blocking certain cookies may limit the functionality of this site.

Do Not Track

This site currently does not respond to Do Not Track signals.


Pearson uses appropriate physical, administrative and technical security measures to protect personal information from unauthorized access, use and disclosure.


This site is not directed to children under the age of 13.


Pearson may send or direct marketing communications to users, provided that

  • Pearson will not use personal information collected or processed as a K-12 school service provider for the purpose of directed or targeted advertising.
  • Such marketing is consistent with applicable law and Pearson's legal obligations.
  • Pearson will not knowingly direct or send marketing communications to an individual who has expressed a preference not to receive marketing.
  • Where required by applicable law, express or implied consent to marketing exists and has not been withdrawn.

Pearson may provide personal information to a third party service provider on a restricted basis to provide marketing solely on behalf of Pearson or an affiliate or customer for whom Pearson is a service provider. Marketing preferences may be changed at any time.

Correcting/Updating Personal Information

If a user's personally identifiable information changes (such as your postal address or email address), we provide a way to correct or update that user's personal data provided to us. This can be done on the Account page. If a user no longer desires our service and desires to delete his or her account, please contact us at customer-service@informit.com and we will process the deletion of a user's account.


Users can always make an informed choice as to whether they should proceed with certain services offered by InformIT. If you choose to remove yourself from our mailing list(s) simply visit the following page and uncheck any communication you no longer want to receive: www.informit.com/u.aspx.

Sale of Personal Information

Pearson does not rent or sell personal information in exchange for any payment of money.

While Pearson does not sell personal information, as defined in Nevada law, Nevada residents may email a request for no sale of their personal information to NevadaDesignatedRequest@pearson.com.

Supplemental Privacy Statement for California Residents

California residents should read our Supplemental privacy statement for California residents in conjunction with this Privacy Notice. The Supplemental privacy statement for California residents explains Pearson's commitment to comply with California law and applies to personal information of California residents collected in connection with this site and the Services.

Sharing and Disclosure

Pearson may disclose personal information, as follows:

  • As required by law.
  • With the consent of the individual (or their parent, if the individual is a minor)
  • In response to a subpoena, court order or legal process, to the extent permitted or required by law
  • To protect the security and safety of individuals, data, assets and systems, consistent with applicable law
  • In connection the sale, joint venture or other transfer of some or all of its company or assets, subject to the provisions of this Privacy Notice
  • To investigate or address actual or suspected fraud or other illegal activities
  • To exercise its legal rights, including enforcement of the Terms of Use for this site or another contract
  • To affiliated Pearson companies and other companies and organizations who perform work for Pearson and are obligated to protect the privacy of personal information consistent with this Privacy Notice
  • To a school, organization, company or government agency, where Pearson collects or processes the personal information in a school setting or on behalf of such organization, company or government agency.


This web site contains links to other sites. Please be aware that we are not responsible for the privacy practices of such other sites. We encourage our users to be aware when they leave our site and to read the privacy statements of each and every web site that collects Personal Information. This privacy statement applies solely to information collected by this web site.

Requests and Contact

Please contact us about this Privacy Notice or if you have any requests or questions relating to the privacy of your personal information.

Changes to this Privacy Notice

We may revise this Privacy Notice through an updated posting. We will identify the effective date of the revision in the posting. Often, updates are made to provide greater clarity or to comply with changes in regulatory requirements. If the updates involve material changes to the collection, protection, use or disclosure of Personal Information, Pearson will provide notice of the change through a conspicuous notice on this site or other appropriate way. Continued use of the site after the effective date of a posted revision evidences acceptance. Please contact us if you have questions or concerns about the Privacy Notice or any objection to any revisions.

Last Update: November 17, 2020