Quick Search Engine FAQs

What are search engines?
How do search engines work?
What is paid inclusion?
What's the most popular search engine?
How much overlap is there?
Do all search engines index the same number of pages?
What is the difference between a search provider and a search engine?
What is "search engine freshness"?
What's a multi-search engine?
What is the invisible or deep web?

What are search engines?

Search engines are tools that let you explore databases containing the text from tens of millions of web pages. When the search engine software finds pages that match your search request (often referred to as hits), it presents them to you with brief descriptions and clickable links to take you there.

How do search engines work?

Most search engines build their databases with programs called spiders or crawlers. These programs roam the Internet and gather resources of all kinds. Individuals can also recommend a site to a search engine. Usually this involves filling out a simple form. These sites ARE NOT screened. The only criteria for inclusion is that they can be accessed. Crawlers (and for that matter individuals) are indiscriminate. Be aware that some of the resources they collect may be outdated, inaccurate, or incomplete. Evaluate your search results carefully.

What is paid inclusion and how does it effect my search results?

With paid inclusion, site owners pay money to guarantee that they will be included and indexed in greater depth than other sites.This is compared to paid placement, which allows you to pay to be number one in any search which includes your subject area. Most search engines offer these options now. Some list them as "sponsored links", but many do not label the results as such. See Buying Your Way Into a Search Engine to find out each search engine's practices.

What's the most popular search engine?

Nielson/NetRatings samples individuals' web surfing habits to determine what search engines are the most popular. Ratings include audience reach and time spent online at each engine. Google currently claims the crown in both categories. See Search Engine Ratings for the latest numbers.

How much overlap is there?

Some, see Thumbshots: Ranking to get an idea of the overlapping and unique hits in some major search engines.

Do all search engines index the same number of pages?

Actually, the sizes of the different search engines vary. Google currently claims to index the most documents with a database of 8.1 billion pages. These pages include PDF files and Microsoft Office documents as well as HTML files. MSN claims to index 5 billion, and Yahoo is close behind with 4.2 billion documents. This is still a fraction of the total number of pages available on the Web, which grows by millions of new pages added each day. That's why its so important to try your search in different search engines. A gem of a result may pop up in one and not another. For more information, see "Search Engine Sizes" from Search Engine Watch.

What's the difference between a search provider and a search engine?

Search providers provide the database of indexed web sites or content to other search engines. For example, Google sells access to its database of web sites to AOL, Excite, and Ask searches. Yahoo!/Overture sells access to its database to MSN, as well as Lycos, Hotbot and Altavista. To learn more about the confusing land search engines, see Who Powers Whom?.

What is "search engine freshness"?

The "freshness" of a search engine is determined by how often it adds new pages to its database, as well as how often it spiders existing pages. See Greg Notess' Freshness Showdown.

What's a multi-search or metasearch engine?

Allows you to input keywords and search a number of search engines all at once. Profusion and Dogpile are examples.

What is the deep web (or invisible web or hidden web)?

The deep web, also known as the "hidden Internet or "invisible web" is that part of the Web which is inaccessible to search engines. It is not so much invisible as harder to find, since it is often in a different format (PDF, Word file, graphic) or location (specialized, subject-specific or subscription databases) than other web pages. Search engines like Yahoo! and Google now include some of these sources in their results.

Maintained by lib-webmaster@wpi.edu
Last modified: Jul 21, 2005, 09:46 EDT