The Open Directory Project - my view

At this stage in the life of the Internet -- yes, it lives and breathes -- we have millions of web pages of all kinds.  About 5.4 million, as of April, 1999.   We have pages like this one that give opinions (everyone is entitled to my opinion!) as well as dirty pictures, home pages, businesses peddling their wares, government offices informing the public, and all sorts of other things.

What makes sense of these millions of pages?  Search engines.

Search engines come in all sizes and shapes.  My search engine page gives a fair cross-section of them.  Search engines make their money by selling advertising.  A popular search engine will get from one to four cents per banner page-view, or about a dollar per "click-through" (that's when the customer interrupts his searching because he finds the banner so fascinating he can't resist clicking it).  So it's in the search engine's interest to be popular so as to attract lots of page-views and click-throughs.  Keys to popularity include

It would be great if a search engine could do all three of these things.   Unfortunately, there are trade-offs.  These trade-offs can best be illustrated by example.  Alta Vista has one of the most comprehensive database of listings.   But they're not very accurate.  Yahoo probably has the most accurate listings, but they aren't as comprehensive as Alta Vista's.

Why is this?

Alta Vista ensures their listings are comprehensive by automating their collection.   They solicit contributions from webmasters, and then use a "spider" to crawl the web, following all links, and index all sites found along the way.  Knowing this, unscrupulous webmasters put all sorts of phony words in their pages, often in hard-to-read colors like yellow-on-white, to increase the probability their page will appear as a result of an unrelated search.  For example, a "Naked Babes from California" site recently included the names of every city in California, and sure enough, it turned up as a result of a search for Palmdale.  This is called "spamming" the search index.

On the other hand, Yahoo pays actual humans to review every web site, and list it with accurate keywords.  This is costly and slow.  As a result, Yahoo's listings are very accurate, but not very comprehensive.  Furthermore, Yahoo has a serious conflict of interest because of the following: starting in February, 1999, for a fee of about $200, Yahoo will expedite the review of your business web site, and give an answer within a week.  In order to make it worth $200 for a business to get an answer within a week, all the free requests must be delayed several weeks, or even months!  Given the pace of change on the Internet (traffic doubles every 100 days!) this approach will surely result in Yahoo's index getting further and further out of date.

How does the Open Directory Project address this trade-off?

The Open Directory Project, http://dmoz.org, enlists the help of volunteers, many of them webmasters, to edit small sections of the directory.   With thousands of volunteers (8,900 as of April, 1999), whose main motivation is to make the index better, the enormous job of hand-sifting all the submissions becomes feasible.   The ODP is not only done by humans, but it is also automated.  Quick and easy tools are provided by Netscape that let editors do their job quickly and easily.  In addition, the leaders of the ODP have written hints for would-be editors suggesting they search Alta Vista to find new links in the categories they edit while at the same time being careful not to steal any portion of another search engine's index.  In other words, editors are supposed to start surfing using Alta Vista, review all new web sites that turn up, and then hand-index them using the automated tools.

It seems, then, that the problem of exponential growth of the Internet that has caused Yahoo to drown -- Yahoo's VP of editorial, Srinija Srinivasan, has admitted they can't hire people at the same rate the Internet is growing (Source: New York Times 4/28/99) -- won't be a problem for the Open Directory Project.  Why?  Because the number of volunteer editors can be expected to grow exponentially, and at about the same rate, as the growth of the Internet.

What keeps the volunteer editor's happy?  "Seeing your name in lights" is one reason -- the editors' names appear at the bottom of the list of links in each category they edit.  Another motivation is to show what a great set of links you can develop to the best web sites.  This is a public version of the "bookmarks" or "favorites" you've been collecting anyway.  A third motivation is just the feeling that you're part of a big project that is growing and becoming more successful right in front of your eyes.  Finally, the editors are pretty well convinced Netscape won't pull the plug on the ODP.

But what if Netscape does pull the plug on the Open Directory Project?  (By "pull the plug" I mean this: declare that it is no longer public, but rather that Netscape is going to own it and sell the living daylights out of it from now on!)

Answer: that's not very likely.  Here's why.  Netscape makes available and licenses the Open Directory to anyone and everyone for free and for ever to do with what they want (like make money off it), except they must do one very important thing: whoever uses the ODP must include a link that encourages users to become editors, and that link must link back to the true original dmoz.org website.  The requirement is very specific -- an object that looks exactly like this must be featured in any copy of the ODP:

This is a key condition, because it ensures that there is just one true original source.  That means that if Netscape pulled the rug out from under the ODP, there would probably be people on the Internet who would take the existing directory (all 22 megabytes of it, as of April, 1999), put it on a server, and encourage all editors to view the new server as the true source of the ODP.

Netscape uses a slightly modified copy of the Open Directory at its site, http://directory.netscape.com/.  Netscape makes money off this copy by selling advertising.  In April, 1999, one of Netscape's biggest competitors, Lycos, made headlines by using the Open Directory -- free of charge -- as a key component of its search engine.  In effect, Netscape is subsidizing its competitor!  Netscape, for its part, didn't complain about this.  In fact, Netscape views this move by Lycos as validation of the Open Directory approach.  It's good to see Netscape living up to its Open Directory strategy.

However I noticed that Netscape doesn't practice what it preaches.  Netscape's own site, http://directory.netscape.com/, does not live up to the letter of the ODP license agreement that requires the exact image shown above -- the little mozilla, the words and the link -- to be displayed conspicuously in any work derived from the ODP.  Netscape's site doesn't feature the little mozilla or the words "Help build...".   Furthermore, the "Become an Editor" link doesn't link *directly* to http://dmoz.org/about.html the way it should.   In addition, the derivative work has been modified in several important ways: banner ads have been added, and the "about.html" file does not include a link to "license.html"  --  in fact, the "license.html" file has been deleted from Netscape's own derivative of the dmoz.org database.  While it doesn't violate the agreement to make such changes, it *is* a violation to make such changes without including a notification to the users of the derivative work.