It took me a while to warm to the term implicit web, but I’ve realized it is a good container for a lot of the improvements in browsing I want to see. I always find concrete examples better than abstract definitions, so I’ll outline a user experience I’d like to build, and how that ties in with the definition.

I have a handful of sites that I visit very often, because they contain trusted information in areas I work on.  I use the term local neighborhood to describe this set of sites. When I do a search, I want these sites ranked very highly, because their results are very relevant to me.

A lot of my friend’s interests overlap mine, and I am more likely to find relevant results in sites they visit frequently. I’d like my searches to rank sites in my friends local neighborhoods more highly too.

This is an implicit web process because the local neighborhood is built implicitly, from monitoring my browsing history, rather than some explicit method such as bookmarking. This is important because almost all users will not take explicit actions, even if they will produce some long-term gain. Technologies that rely on users doing something that feels like work end up stuck in a geek ghetto.  Nobody I know who works outside IT, uses del.icio.us.

General users are more willing than us to surrender some privacy in return for improved features. me.dium relies on this. I’m trusting them with my entire browsing history, and in return they give me information and communication about other users in my current local neighborhood.

Google is moving in the direction I want with its search history feature, but that only biases sites that you find through searching. I go to my local sites through typing the first few letters in the address bar, so they won’t be included. Flock has a search history feature too, that looks through the pages you’ve browsed recently. This is closer, but your history is kept locally, so you easily lose it if you move machines, reinstall, etc.

Neither of these approaches work with my friends’ local neighborhoods. There’s a serious obstacle to this happening; the information about my social graph is stored in a database owned by a different company than the one I use for searching. At the moment, information like this is only ever shared within an organization, since it’s treated as valuable and proprietary. This sucks for users, since they created a lot of the information themselves, and they can see it all, so why can’t their software?

Aggregators try to get around this by having users give them their passwords and user names, and screen-scraping using a central server. This is both fiddly for users to set up, and easy for the providers to block if they want. A few services also offer a web API, but these are fairly limited in the information they provide, and subject to being blocked by the providers at any time. The fundamental conflict is that the value of these companies is largely based on their data, and they aren’t about to give it away to a competitor.

A better way to solve this is with the semantic web. The idea is you make web-pages understandable by software, not just human readable. As you browsed Myspace, client-side software would interpret each page and discover who your friends are, and the more important ones whose profiles you visit, or who you exchange messages with.

Sounds great, but so far it’s been a pipe-dream, because it’s like web APIs, there hasn’t been a good reason to make your pages easily understandable by third-party software.

One of my big research efforts is finding some simple, practical ways of jump-starting this process, by using simple rules to reliably work out semantic information about a web page. Google Hot Keys is one result of this work; it analyzes pages to work out which are search results, what the search terms are, which links are the pages associated with those terms, and what the ‘next page’ link is. It’s promising to see that the rules are robust enough that they work with over 40 different foreign language Google sites, as well as Ask and Live.

It seems to me that the only way to fulfill the promise of the implicit web is to combine client-side technologies that have access to all the information a user does, and software that can pull data directly from the web-pages as they browse.

Funhouse Photo User Count: 944 total, 63 active. A bit of a larger total increase, still not very exciting.
Event Connector User Count: 5 total. The adwords campaign has gained me one user, got 16 clicks, and cost $1.13 so far. Probably time to change tactics!