The Invisible Web
Learn about:
The Invisible Web: major components, size, searching, and selective search tools.
What is the Invisible Web?
The invisible web is the portion of the web that is not searched by general search engines. It is sometimes referred to as the hidden or deepweb. As the amount of information published on the WWW has grown, web site developers have turned to databases as a means of managing the increasing amounts of data. These databases are searchable with site-specific search engines, but are not searched by general search engines (individual or meta-searchers).
Although the invisible web has gained attention recently, it is not a new phenomenon. You have already searched portions of the invisible web when you've accessed library catalogs or information services, such as OCLC FirstSearch, Databases for Alaskans, Cambridge Scientific Abstracts, etc.; read the New York Times online; used an online calculator; or viewed a "page image" document. These are sites or pages that are restricted to login, are fee-based or subscription, are protected by firewalls, are generated in response to a query, or cannot be searched because of format.
Major components of the invisible web include:
How Big is the Invisible Web?
A recent study,
BrightPlanet's white paper (optional reading)
indicates that the invisible web is 400-550 times larger than the traditional
(surface or open) web. They estimate that there are more than 7,500 terabytes
of information contained in the invisible web, compared to the 19 terabytes
of the surface web. Most of these sites are accessible to the public,
free of charge.
Searching the Invisible Web
Companies have been developing search engines to search the invisible web and are now making these tools available. Rather than retrieving web pages or documents, invisible web search engines direct the user to an appropriate searchable database. Some even generate a search form for your convenience.
Searching the invisible web is actually a two-step process.
First, locate the appropriate database to search. Browsing is the easiest method. Invisible web search engines are accompanied by a categorized list or subject directory. Browse the appropriate category to make your selection. Or, you can use keywords, including natural language, to search for a database using the search engine. Searches should be kept simple.
Second, search the database you just located. Review the HELP screens for tips on improving the effectiveness of your search query.
Selective List of Invisible Web Search Tools
Complete Planet ( http://www.completeplanet.com/index.asp)
Direct Search ( http://www.freepint.com/gary/direct.htm )
ProFusion ( http://www.profusion.com )
FirstGov ( http://www.firstgov.gov ) -- access to federal government databases
Evaluation of Invisible Web Tools
Invisible web tools are still evolving and like other resources, should also be evaluated. To locate evaluations try the following resources:
Additional Resources
The Invisible Web - Tool Kit for the Expert Web Searcher
Material may be quoted or reproduced for educational purposes without prior permission, provided appropriate credit is given. Any commercial use of the material is prohibited without prior permission from the Rasmuson Library.
Created: 08/01, S. Hahn (ffslh1@uaf.edu)
Updated: 09/03
