This ZDNet article really reminded me why I started PeteSearch. It’s from 2004, and laments the lack of progress in searching, and not much has changed since then! Sure, there’s been iterative improvements, more flexible term matching, search history and the like, but nothing that a Google user from 2000 would be surprised by.

Part of the problem is that Google can copy most foreseeable outside innovations if they turn out to be popular. It’s really hard to make a business case for funding a company that would in effect be providing them with free R&D, with little prospect of a return. Google themselves are experimenting with new models, but without real competition, they’re in no rush to cook their golden goose. Ask have been the most innovative with their search UI, but even that is still based around the same basic layout.

One phrase really struck me from the article: "Don’t expect users to apply more than the basic tools and techniques to acquire information from a search engine." The stats show that only three percent of people used quotes or other advanced syntax.

A lot of people have concluded that this means people don’t want anything better, and there’s no point trying to improve the page-of-links presentation of results. My firm belief, and the reason I’m experimenting, is that I think it’s just a local-maximum in the space of possible UIs. Why not have a grid of 25 thumbnails, with the positions of the terms marked on each? Or live snippets of the actual rendered page below each link, not just the text? Or a micro-fiche-style view, where you cycle through all the pages at speed in full-screen? Sure, these are random examples, but are people in 2030 really still going to be using plain pages of links?

PeteSearch has finally made it through Mozilla’s review process, and is now on the main add-on site. It’s great to see it on such a high-profile site, but one of the really nifty things is that you now don’t have to go through the two-step process of adding petesearch.com as a trusted site in Firefox to install the addon, since mozilla.org is trusted by default.

It was a long and rocky road getting approval, since they recently added a new sandbox system, and require user reviews before they’ll allow an addon onto the public site. This does make sense, but unfortunately there’s almost no users writing reviews in the sandbox, I think at least partly because it’s tough to get to even if you know about it, and almost impossible to discover if you don’t.

Luckily Pavel Cvrček, Mike Shaver and Shawn Wilsher came to my assistance, and helped me work out how to get the user reviews from news sites and blogs taken into account when they’re evaluating my addon for publication. Shawn has a post explaining the policy in more detail, but in simple terms you need to add links to the external reviews in the  ‘Notes to Reviewer’ section before you nominate it. You get to that section by clicking on the  version number link, eg


Thanks Pavel, Mike and Shawn, I really appreciate your help!

PeteSearch is a semantic web application, it’s taking web pages designed to be read by humans and turning them into data that can be processed by software. It’s a pretty specialized application, focused purely on pages that list external sites associated with particular search terms, but the wide range of sites I’m able to support using the same code shows that my approach is robust.

The model I use for search pages is that they must contain three pieces of information:

• A list of search terms, embedded in the URL
• A list of external sites associated with those terms
• A link to the next page of results

All of the recognition of these is data-driven, using a definition for each engine that includes

• What is the host name and action used by the engine, so we can tell what’s a page of search results. For google that’s google.com/search
• What precedes the search terms in the URL, eg for google that’s q=
• Which external sites are linked to, but not part of the results, eg google links to answers.com for definitions of words
• Which words indicate an external link that isn’t part of the results, eg google links to the cached results on numbered servers using Cached as the link’s text
• Which word is used for the link to the next page of results. For English that’s almost always Next but I also support other languages

You can experiment with this by editing the SearchEngineList.js file inside PeteSearch, it contains an array of these definitions, and an explanation of the exact format they’re stored in. It’s pretty straightforward to add a new engine that fits into this pattern, and most of them do.

The only way that the semantic web is going to progess beyond proofs of concept is if there’s some concrete, practical and commercial application for it. I’ve seen this with AI, its applications in robotics and games have pushed the field forward much more than pure research. The semantic web is stuck in a chicken-and-egg situation; nobody builds applications because nobody builds sites that are data sources,
because nobody builds applications, etc.

I’m not the only one to notice this, Piggy Bank is an MIT project that’s much more ambitious, and works like Grease Monkey in that it provides a framework to support data capture from many different sites using plugin scripts.

My goal is to demonstrate that a semantic web application can be useful today, in the real world, by creating a compelling tool based on my approach. I’m worried that unless somebody can show something useful, it’s going to succumb to the AI curse and remain the technology of tomorrow indefinitely!