Search upstarts storm Google’s gates

Re:Library Search Engine

That’s a good idea, but if you want something Google-like but original, you will probably need some real computer science expertise in the areas of natural language processing and semantic analysis (which the library world thus far hasn’t seemed to attract), as well a substantial number of servers and storage arrays, and a big, big pipe. Matching the scope of a Google or a Yahoo probably requires retrieving and indexing a hundred or more web pages per second, and if we don’t match that scope and at the same time do something unusual, why will anyone bother to switch to our search engine? It isn’t simply a matter of setting up Swish-E or even Lucene and turning it loose. Behind Google’s simple input box are subtleties undreamt of in our philosophies.

The strengths of libraries have always been in (a) the expert selection of materials, (b) the expert organization of the same, and (c) the expert assistance with using the same. It’s not clear to me how we translate those strengths into the search engine world in a way so compelling as to put librarianship at the forefront of Joe and Jane User’s daily information usage. lii.org, which is a directory rather than a search engine, seems like the natural way to translate the traditional virtues of librarianship onto the web. It’s a great resource, and doubtless the homepage for many a refdesk computer, but in the end what will make it more compelling than Google’s simple box into which one types in a few words and gets back a generally useful list of hits? We know that we can do a better job than computers (gotta see Desk Set again!), but the search engine often does a good enough job and has millions and millions of documents at its immediate disposal.

So do we continue with what we’ve always been good at and hope that if we build it they will come, or can we find some way to relate what we’ve always been good at to the search engine world? I wish I had an answer.