Simple Search page users just type a word or string of words and AltaVista Search returns a list of reasonable hyperlinks to Web pages. Many users cut and paste a full sentence (or sentences) into the dialog box window and let AltaVista Search find all the Web pages that correspond to the query. AltaVista Search's collection frequency rating features automatically rank the returns.

Advanced Search page users use Boolean, specific Results Ranking Criteria, Start Date and End Date instructions to control returns. This allows experienced search-and-retrieval users control the returns delivered by AltaVista Search. Researchers and librarians often prefer AltaVista Search's Advance Search page.

Most users usually get the best results by using Simple Search and letting AltaVista Search's collection frequency rating software control the query.

Robots.txt is an ASCII file placed in the Web server's root directory that tells Web crawlers whether or not they are invited into the Web server's Web site. Robots.txt can forbid entry to crawlers. It can also direct crawler activity to specific sections of the Web site.

Unfortunately, not all search engines honor the Robot Exclusion Standard. Verity based crawlers for example can be told to ignore Robot.txt instructions forbidding entry onto a Web site. Aside from being ethically questionable, this can have dire consequences for the server getting returns from the errant crawler. Some Webmasters routinely send a few thousand large e-mail messages to the server that sent out the crawler that ignored their Robot.txt instructions. This issue may not be important to Intranet Web server situations, because the company can set internal policies to control which Web servers are crawled.

AltaVista Search always honors Robot.txt instructions. Webmasters, Web consultants and Web magazines say AltaVista Search uses a well mannered, polite crawler.

Database Integration

The AltaVista Search Developer's Kit 97 is available for Windows NT and Digital UNIX developers. It is a toolkit that enables system integrators and developers to integrate AltaVista Search with structured and non-structured databases. AltaVista Search Developer's Kit (SDK) 97 search-and-retrieval solutions are extremely fast, efficient and scaleable. Features and capabilities include:

• Toolkit to index structured and non-structured databases.
• API to AltaVista Search's NI2 C library.
• Works with any language that links with C.
• Developers build integration software between AltaVista Search's inverted word index and data repositories.
• Full documentation with coding examples.

AltaVista creates an "inverted word index." The ranking is based on the collection frequency weighting. AltaVista Search's collection frequency weighting lists the best hits first and reduces the number of junk hits.

AltaVista Search's index contains information that describes the structure / organization of the documents in the index. HTML files use meta data tags to describe the documents structure. AltaVista Search uses the meta data tags when users perform fielded queries. Examples of fielded queries include searching for titles, URLs, and links.

AltaVista Search Intranet eXtension 97 also allows users to perform Boolean logic and wild card queries. This makes it easy for users to easily expand and control their queries. For example, searching for Baltimore AND hotel and using the word hotel in the relevance ranking dialog box will put 10 very useful hyperlinks to specific hotels in Baltimore, MD at the top of the returns list.

Stemming

Stemming converts words to their root and the search is done on the root and all derivatives of the root. For example, if you are searching for the word "systemic", stemming search engines would recognize that its root is "system" and then it would search for all words that have system as the root.

This means you would receive returns on: system, systematic, systematical, systematically, systematics, systematism, systematist, systematization, systematize, systematized, systematizer, systematizes, systematizing, systemic, systemically, systemization, systemize, systemized, systemizer, systemizes, and systemizing.

Stemming causes lots of false hits because it automatically looks for too many options. On the other hand, it also makes it very easy to find closely related words.

Excite, the Microsoft Index Server, Verity and OpenText all utilize stemming.

Thesaurus Techniques

This is another language-dependent technique designed to automatically expand the user's search criteria. For example, if you are searching for the word hat, a thesaurus-based search engine might expand your query to include: cap, bonnet, sunbonnet, and hood. This can be a useful technique when you need to expand your search and you are not well versed on your topic. It can also result in false hits when the user is looking for a particular word or phrase.

Verity and OpenText utilize thesaurus techniques.

Morphological Analysis

Morphological analysis is another common querying technique used by older search engines and some of the new Web-based search engines.

Sentences and words are parsed to determine their meaning / purpose / structure. This process is language dependent. Every language you need to work with requires extra software.

Concept Techniques

This technique is also known as "words commonly seen together." This technology requires a relatively large set of Web documents to provide good returns. Excite uses this technology, and anecdotal feedback says few intranets currently contain enough textual data to give consistently accurate returns.

Excite continually refers to their use of concept techniques as a major plus point. However, here is the rest of the story. The Excite software does not index the entire Web page. Instead, they only index the "remarkable" words.

Acronyms with periods are a problem for many concept-based search engines.

Concept engines do not handle phrase searches. For example, try searching for "to be or not to be" on Excite's World Wide Web site. You will get few if any returns because prepositions are not "remarkable" words. (By the way, AltaVista Search found over 2,000 documents with the phrase "to be or not to be" on the World Wide Web at the time of this writing.)

Suffix Trees

Suffix trees allow users to search for parts of a word or a phrase just as easily as they can search for a single word.

Suffix trees also have some serious disadvantages. The database / index is usually as big as the original text. They also need to have the original text online. This means the overall database is twice as big as the information repository being indexed. Constructing the suffix tree oriented index is a much slower process than constructing an inverted index based on words. Performance often suffers while the index is being updated. AltaVista Search's inverted word index does not have these problems.

To sum this up, most users are not looking for parts of a word when they do a query. Most users also feel that phrases, single words and wild cards provide plenty of querying flexibility. The ability to look for a piece of a word is a minor advantage when compared to the major disadvantages around index size and indexing speed.

Virtually all our competitors push query-oriented user interface benefits. They try to convince people that language based queries are the best way to find information. This is not hard to do because people have been using language-based querying techniques like word lists and thesauri for years. Our competitors' user interfaces are optimized to make language based queries as easy as possible.

What they don’t reveal are the limitations of the indexing technologies that lie behind their user interfaces. Our competitors don’t mention scalability, indexing efficiency, index completeness, query response time and language independence features, because AltaVista Search is the clear winner in those categories, the most important measurement criteria.

Weak Points

Competitive products do not scale well. For example, Microsoft Index Server, Excite, and Verity based solutions are often sold as Internet appliances. You must keep adding new search servers as your user community and information repositories increase. Then, you daisychain your queries from search server to search server. This means longer response times. You also get to buy, install, and maintain more search servers.

Open Text and other search engines that rely on suffix trees can handle large indexes, but performance and disk space requirements are often major issues.

Concept-based search engines such as Excite do not index every word on a Web page, and they throw many words away. They also need a large Web site to consistently return accurate query results.

Verity and OpenText crawlers are not as efficient as the AltaVista Search crawler. We find data faster.

Their indexes are not as efficient. AltaVista Search's index is smaller and more efficient than any of our competitors.

Their query engines are often language dependent. Adding additional languages means additional money.

Pricing

This illustration shows where the three paid-for products fall when you graph price against functionality, and functionality is weighted towards scalability and large deployments.

Functionality, weighted Towards Scalability and Massive Deployments

Verity handles large search-and-retrieval needs by daisychaining search servers together. This illustration is based on two of their products: their Information Server and their crawler. The Information Server, also known as the Topic Server, lists for $8,000 when it is installed on a single CPU server. Their crawler lists for $10,000 and it will crawl 10 sites. The crawler can be extended to 10 additional sites for $4,000. If you need to crawl more sites, you keep buying $4,000 crawler extensions.

OpenText sells two versions of the same product. Their 64 bit version costs more than their 32 bit version. OpenText is no longer competing for search-and-retrieval business. They are concentrating on document management and business process reengineering sales.

The best World Wide Web search engine is now available for private information. Covers Web- and LAN-server information repositories. Indexes over 200 file formats. The crawler is included at no extra cost. Optional database integration software is available. It is very scalable.

Large multinational intranets are easily supported with a centralized search server. Statistical indexing and querying is language independent. Indexes every word, every number, every phrase - all content is important. Nothing is ignored or left out of the index. Language independence. Fast, efficient indexing and querying

Best suited for:

• Companies who expect considerable growth for their intranet.
• Companies who have already tried Verity and other search-and-retrieval products and found that they couldn't scale to meet their needs.
• Companies with large database repositories that want to improve search-and-retrieval performance.
• Multinational organizations.
• Organizations who want to decide for themselves how to distribute the search-and-retrieval function as opposed to letting the technology make the decision.

Excite doesn't index Office documents. It only supports HTML and text. It doesn't handle databases or PC application files. Its concept and relevance ranking software needs a large Intranet to give accurate query results. It only indexes remarkable words. This keeps the index size down, but it means exact phrase search is not possible.

Best Suited For:

• Companies who see no need for LAN-based search-and-retrieval.
• Companies who do not need to look for PC application files.
• Companies who have moderate querying requirements.

This is a single-server application. It does not crawl its way through an intranet. It only indexes files within sub-directories on a single NT server. If your Web pages are located on multiple Web servers, this solution will not work well in your environment. If your Web pages contain hyperlinks to Web pages residing on other Web servers, the Microsoft Index Server will not be able to index all your material.

If you use applications from many different vendors, the Microsoft Index Server will not be able to index all of them. This is very much Microsoft-centric software.

It only searches in 7 languages. Non-remarkable words are deleted from the index. This means exact phrase searches are not possible.

Best suited for single servers.