I’m a firm believer in studying the techniques developed over centuries by librarians and other traditional information workers. One of the most misunderstood and underrated processes is indexing a book. Anybody who’s spent time trying to extract information from a reference book knows that a good index is crucial, but it’s not obvious the work that goes into creating one.

I’m very interested in that process, since a lot of my content analysis work, and search in general, can be looked at as trying to generate a useful index with no human intervention. That makes professional indexers views on automatic indexing software very relevant. Understandably they’re a little defensive, since most people don’t appreciate the skill it takes to create an index and being compared to software is never fun, but their critiques of automated analysis apply more generally to all automated keyword and search tools.

• Flat. There’s no grouping of concepts into categories and subheadings.
• Missing concepts. Only words that are mentioned in the text are included, there’s no reading between the lines to spot ideas that are implicit.
• Lacking priorities. Software can’t tell which words are important, and which are incidental.
• No anticipation. A good index focuses on the terms that a reader is likely to search for. Software has no way of telling this (though my work on extracting common search terms that lead to a page does provide some of this information).
• Can’t link. Cross-referencing related ideas makes the navigation of an index much easier, but this requires semantic knowledge.
• Duplication. Again, spotting which words are synonyms requires linguistic analysis, and isn’t handled well by software. This leads to confusing double entries for keywords.

A lot of the really interesting services out there are about discovery, rather than search. Since they’re both ways of finding content, it’s worth looking at what makes them different.

Search is goal-oriented, discovery is about the journey. It’s the difference between going to the hardware store to get a Phillips screwdriver, and browsing the travel section in a book store. Rather than having very specific criteria in mind for what you want, you’re using indirect clues to help you find something that will meet your general needs. For a travel book that might include whether you’ve seen it mentioned in a review, if you’ve enjoyed the author’s work before, if it has an attractive cover, if there’s praise from people you trust, if your hairdresser mentioned the location, if you’d seen a documentary on the area, or if it happened to be sticking out from the shelf a little more than the others.

Search is solitary, discovery is social. Most of the factors behind buying a travel book are about interactions you’ve had with other people. Digg is one example of trying to emulate some of those traditional mechanisms for finding popular items. Facebook’s addictiveness is all about being tapped into the pulse of your social circle, not with a particular goal in mind, but just to keep up with the context and doing the equivalent of picking fleas off each other’s backs (now that’s a Facebook App idea!). The power of me.dium comes from the injection of social context into browsing. It restores the cues we’re used to in the physical world, so we can judge locations by seeing where both our friends and strangers hang out.

Search is about universal answers, discovery is customized. Though Google talks about searching the sites your friends frequent, I think that functionality will be much more useful for discovery. It’s not likely that your social circle will be visiting the most authoritative sites on all the specific questions you’ll want to get definitive answers on, the sample size will just be too small. Instead, finding out which sites on general topics are popular amongst your circle would be a lot more interesting. Discovery is a lot more about your personal taste, and that’s something you likely share with your friends a lot more than the general population.

Discovery is a background task. Often you’ve got some general interests that you want to know more about, but you’re not actively taking steps to find out more. Instead you’re keeping an ear to the ground while you get on with other activities. This is where applying some social network techniques to the workplace can be really interesting. Seeing updates on what your colleagues are up to will often trigger some thoughts or connections on topics you’re interested in, and lead to discussions you wouldn’t have had otherwise. That could be the real killer app for social networks in the enterprise.

For more thoughts on this, there was a fascinating discussion over on the Lightspeed blog a few months back.

ajax  api  bho  c  crossloop  crossloop review  cruz island  dom  dom in  dom in c  error  example  facebook  facebook api  facebook footprints  find self  firefox  heat load dlls find  hiking santa  hiking santa cruz  hiking santa cruz island  how  how to  ie  ie dom  ie dom in  ie dom in c  in  in c  jolla valley  jolla valley campground  la jolla  la jolla valley  la jolla valley campground  managedq  outlook  outlook api  review  santa cruz  santa cruz island  search  server  socket  socket server  the  to  valley campground  wix  wix heat load dlls find  write 

After using statcounter to track around 800 visits, I’ve put together a search cloud for this blog.

I’m still getting a lot of hits for my early review of Crossloop, thanks largely to the success that Mrinal and the gang have had with their free screen-sharing tool. I’ve seen a lot of people looking for more information on Facebook programming, that’s shown pretty clearly in the cloud too. I get the feeling that most developers are still fairly cagey about sharing information, or just plain too busy to post the collection of tutorials and tips that usually grows up around a platform.

For the Outlook API, it’s almost the opposite problem. There’s loads of resources out there, many going back to the mid 90’s, but almost nothing that gives a broad introduction to all the different technologies you can use to work with Exchange and Outlook.

It’s been great to see my local camping guides reaching so many people too. It’s a real long-tail endeavour, but it’s a good feeling to know that anyone looking for camping spots in the Santa Monicas can now find out on the web, whereas before the information just wasn’t there.

As I gather more statistics, it’ll be interesting to do some per-page clouds. Since the whole blog covers a range of subjects, it’s hard to get an idea if you can get good hints on its meaning from this cloud. Looking at visits to particular pages more focused on a topic would be a better test.

If you want to know which search terms are most likely to find your site, I’ve uploaded a PHP library that creates search clouds from your logs. To use it include searchcloud.php and call create_search_cloud(), passing in the location of your log file, the name of your site, the number of tags to produce and the min/max font sizes in percentages. You’ll be returned a string containing the HTML for the cloud. Here’s an example:

echo create_search_cloud("visitlogs_petewarden.txt", "petewarden.com", 50, 50, 250);

You can see it working on this example page based on statistics from my old open-source image processing site, which I’ve also included with the library for testing purposes.

Based on the examples I’ve tried, my hypothesis that the most frequent search terms are a good approximation for the meaning of the site holds up. If you take the top 8 terms from the petewarden.com cloud, you get "after effects", "plugins", "effects", "after", "how to", "install". "how to install", "petes plugins". 4 of them would be good tags or taxonomy categories for the content, and on inspection the use of more sophisticated rejection of duplicates and stop words would help increase that ratio. I’ll be interested to hear how this works on some of your sites.

A search engine is designed to take some keywords, and return web pages that match them. What fascinates me is that that mapping of words to pages could easily be done in reverse. Given a particular web page, tell me what keywords are most likely to find it. My hunch is that that set of words, maybe presented as a tag cloud, would give a pretty good summary of what the page is about.

The closest example I’ve found out there is this blog entry. It’s got what appears at first to be a fairly random list of keywords, but digging into them, it looks like Darrin is a Vancouver-based Titanic fan who’s posted about the beautiful agony art project and has done a lot of wedding posts.

What’s really interesting about this is that the search terms that show up aren’t just based on textual frequency within the site, they’re also the product of how often people search for particular words at all. Essentially it’s giving a lot more weight to terms people actually care about, rather than just all terms that are statistically improbable.

At the moment the only way to implement this is to process an individual site’s visitor logs to pull out the frequency of keyword searches that lead to a visit. However search engines know the historical frequency of particular queries terms up front, so it would be possible for them to take an arbitrary new page and simulate which searches would be likely to land on it. You could do something similar for a mail message, essentially you’d be filtering statistically improbable phrases to get statistically improbable and interesting phrases instead.