The Past, Present, and Future of Online Directories

05/11/04

Building Blocks of WWW Navigation

Years ago, inclusion in the top online directories was a necessity for anyone interested in receiving traffic to their new website. Yahoo, About, Looksmart, and The Open Directory dominated the market by providing the web's most reliable mines of resources. These human edited, hand selected directories of categorized websites seemed as if they would ever be the easiest way to sort and find online information.

Internet Growth - An Issue of Scalability

But then something happened… buying, building, and managing websites became both easy and inexpensive. Everyone and their dog had a website in some capacity, whether it was a personal homepage, e-commerce store, or internet portal. The internet grew too fast for the directory structure to keep up with.

Even if the directories' editors could manage to review and include the hundreds of new sites being introduced every day, an all inclusive directory would have far too much content to efficiently display. A user might be presented with a page of thousands of links in a single end category.

Such was the case with the Open Directory. Founded on the "open source" philosophy - data and technology created by all, refined by all, and free for all to use - the Open Directory continued with its free inclusion model, maintained by a number of volunteer editors. As submissions poured in, the directory grew unchecked. Categories became so large that editors could no longer manage the existing content, slow review times left the directory outdated, and few resources remained for auditing inaccuracies and broken links. Making matters worse, additional technology costs accrued as the need for storage space and bandwidth increased. The result - a slow, partial directory of broken links and unorganized content.

The fundamental dilemma presented by WWW growth would lead to two unique developments in the directory market… the paid inclusion model and the necessity for search technology.

Paid Inclusion is Born

In order to meet the demand for submissions, some directories chose to being charging a fee for inclusion. Yahoo and Looksmart each went this route. By requiring a nominal fee, webmasters with sites of little content or value would not bother to pass their properties across the directories' editors' desks. The primary benefit of this system is that it allows the editors to focus on those sites more likely to be worthy of inclusion. The drawback is that editors may never have the opportunity to include sites which offer great value to users but either have no budget or no initiative to pursue a paid listing.

The Improvement of Internet Search

Along with the refinement of internet search came the downfall of the directory. Initial search technology was slow, inaccurate, and based on a small sample of what was available on the entire internet - these first "search engines" posed little threat to the directory structure. Eventually, though, the system where relevant results could be indexed and delivered without human intervention proved to be much more scalable and cost effective than the directory system.

Search technology has developed by leaps and bounds in the past several years. Results are faster, more accurate, more comprehensive, and filtered for suitable content, making the old directory structure obsolete.

What Becomes of the Modern Directory

Today, the most successful directories are those which are based on a pay per click model. Such directories include demographically-targeted Business.com and Verizon's SuperPages. In these, and many other PPC directories, any site is eligible for instant inclusion in a categorized directory of websites, products, and services.

Automated submission and bidding systems allow for the efficiency of search indexing, making these directories more scalable than their human-edited counterparts. Because listings are paid on a per-click basis, the categories are inherently more refined and listings are seldom out of date - advertisers have no interest in bidding for untargeted visitors.

Though highly relevant, the costs of inclusion in a pay per click style directory can be high and only the most efficient, highly profitable, or niche-targeted sites can afford to remain.

Getting Local

Perhaps the best chance for the Directory structure's continued existence is in presenting local information. Whereas search engines are able to provide results for websites, human-edited directories have the ability to include content which may not otherwise be online. This system may not be efficient for global searching, however, regionalization is highly targeted and a single editor can easily identify and include locally relevant information within the contents of his or her directory branch.

A first mainstream application of this concept is Yahoo's GetLocal Yellow Pages.

An advanced application of this concept is the "SmartView" technology integrated into Yahoo Maps. Upon viewing a map of a town, city, state, or region, a navigation menu allows the user to select from a number of businesses and services which are subsequently displayed as icons on the map itself. Roll your mouse over the icons, and the name, address, phone number, and a link to driving directions are displayed. "Directories" of this nature are likely to dominate the future of local "search".

Conclusion

Without including all reputable sources in a given content area, a modern research tool is virtually useless. Herein lies the primary obstacle facing online directories - without automated systems in place it is virtually impossible to efficiently and effectively reference all online resources.

There may never be a machine substitute for the human being, but today there are plenty of applications capable of outperforming the human-edited directory.