With Defrag fast approaching, I’ve been spending some cycles thinking about what the Implicit Web actually is, and where it’s going. When I’m staring at this sort of problem, a technique I find really useful is "stamp collecting". Gather as many examples as possible, list their important properties, group them into clusters and look at what patterns emerge.

Here’s my current list of Implicit Web services currently out there, with a couple that are on the borderline of the term. I’ve not got enough to meaningfully group them, so they’re alphabetical:

Adaptive Blue – More of a semantic web application, but they do offer a Firefox extension. Being client-based is a distinguishing feature of implicit web apps, since that’s the only way I know to get access to the user data needed.

Amazon – Their recommendation system is the grand-daddy of a lot of the apps that take raw information on user’s behavior, run some magic algorithms, and return something useful back to the customer. It’s a hard trick for most startups to repeat, since almost nobody has access to the Amazon’s breadth of data. This is why client-based solutions that can track behavior across many sites seem like the only practical solution.

last.fm – A true implicit web app, they have client-based tracking of the user’s behavior, they piggy-back on other people’s applications to gather their data and use that to return Amazon-style recommendations. It does make me wonder about the ‘web’ part of the term though, since that seems to imply web browsing. Maybe ‘implicit internet’ would be more appropriate?

me.dium – Another app that fully fits the term. A unique feature is that they use the social graph to combine information from multiple users, which I think is a very promising area for implicit web applications. Being able to pool data from your friends is a great way of discovering relevant new content.

MySportsNet.ca – This is one I came across relatively recently. It’s a client-side app that monitors your browsing, and tailors a sports portal site to match your interests based on that data. What’s really interesting is that it’s aimed at a mainstream audience of sports fans, rather than geeky early adopters. I know from my game career that the sports audience is massive, and willing to pay for something ties into their passion, so I’ll be following its progress closely. The only audience I know that’s similar is music, and it’s relevant that the most successful implicit app so far, last.fm, tapped into that demand.

tape failure – This is a service I’ve only read about, but unfortunately their site seems to be down at the moment. They’re not an implicit web app at all, but it does seem like they have a good solution to the browsing data collection problem.

Let me know if you think I’m missing any. I may put together a page tracking new services, since I think we’re going to be getting a lot more over the next year.

Funhouse Photo User Count: 1,916 total, 92 active. The proportion of profile-box adds was a bit higher this time, which is promising because it scales a bit more virally than the product directory.

Event Connector User Count: 84 total, 9 active. Not much happening on this front.

I was a console programmer for six years. Games are the only pieces of software that people use purely for the joy of interacting with a computer. There’s no reason to play, except to have fun.

This means that the user interface is crucial. With other software, people will put up with the pain of a bad UI because they’re trying to accomplish some real-world task. If a consumer picks up a video game and it doesn’t let them have fun within a minute or two, they will give up on it. The interface has to be easy and fun. It can still be deep, but that complexity must be intuitive and discoverable, and not presented like the Space Shuttle’s control panel.

What really excites me about the implicit web is the promise of using the gathered data to turbo-charge everyday interfaces. A simple example is Firefox’s address bar; it remembers the URLs I visit, and when I start typing a new one, the suggestions are in most-visited order. By contrast, I wouldn’t class Google Suggest in the search box as an implicit service, since it doesn’t customize the suggestions based on my behavior, and it’s a lot less useful for me.

When I was working with Nintendo, the holy grail was the ‘one button game’. Think Mario 64, where you managed complex interactions with a 3D world mostly with the joystick and a single button to jump. Stumbleupon is the web service that’s closest to this, I’ve heard it described as the ‘Forward button‘ for the web, and it really delivers a lot of value with very little input needed from the user. Google Hot Keys is my attempt to move searching in that direction, though there’s no implicit component.

One of the parts I’m most anticipating about Defrag is seeing all of the innovative interfaces that the teams will be showing off. There’s so many possibilities for improving the user experience, I can’t wait to see what people are coming up with!

Funhouse Photo User Count: 1,817 total, 93 active. Still growing steadily, but slowly, with most of the additions coming through the product directory.

Event Connector User Count: 77 total, 4 active. Still working with conference organizers, not much to show yet though.

Implicit web apps rely on access to information you don’t want everyone to know. Unlike traditional server driven web sites, implicit web apps often run on the user’s own machine. This gives them access to all the user’s data, whereas a web service can only see a small slice covering what the user did whilst visiting that site.

The only difference between an implicit web client app and spyware is intent. Fred Wilson has a quote "If someone’s going to spy on you, it’s probably best if it’s you." I think "If someone’s going to spy on you, it’s probably best if it’s us" is a better reflection of the current state of the implicit web scene. We aren’t empowering users by letting them own their information, and control exactly what is revealed. Instead at install time we’re asking them to sign over the right to pull all their information onto our servers.

This isn’t a big issue yet, because there’s not much awareness amongst users of the dangers. But it would only take one big privacy breach to start people worrying. We need to plan ahead to make sure we don’t get classed as spyware by zealous blockers.

I think the model for the future is something like the Attention Trust. Set up to provide a standard for the treatment of user’s web-browsing behavior, they mandate a set of principles their members must follow. In return, organizations that meet those principles can display a badge demonstrating their trustworthiness.

It’s not perfect, there’s not a rigorous inspection or application process to join, it’s mostly self-regulated, and the rules are focused on web-browsing. But it is an organization I expect to grow and mature as the demand from legitimate implicit web companies to avoid being labelled as spyware gets stronger. They also offer a very interesting Firefox extension for tracking user’s web-browsing, I’m tempted to try a port over to IE.

The trickiest practical part of this is that providing the sort of fine-grained user control will take a lot of extra engineering, and some smart UI to avoid baffling the user with a space shuttle control panel of options. Most services allow you to temporarily disable information capture, but I think one of the requirements is going to be the ability for users to remove data from your server after it’s been captured, and that’s going to be a lot harder to implement.

As I was researching this post, I ran across an article by Alex Iskold on ReadWriteWeb that was really helpful. I guess I wasn’t the first to spot Amazon as the ur-Implicit-Web-App!

Funhouse Photo User Count: 1,686 total, 73 active. The stats have moved at least, but still seem a little flakey, showing data from three days ago.

Event Connector User Count: 65 total, 10 active. No progress on signing up a conference, I will be chasing this up again, and considering some different approaches to reaching organizers.

"It looks like you’re writing a blog post. Can I Help?"

Clippy was an implicit non-web application. He’s was built on a really clever piece of implicit analysis technology, and deliver a nightmarish user experience. It was so bad, there’s even a research paper devoted to exactly why he was so hated.

I try to remember Clippy when I find myself getting too deep into arcane algorithms, and too far from the user experience. One of the best pieces of advice I’ve had about application design is to imagine an actual concrete user in as much detail as possible, including a name. Then describe, step-by-step, her thoughts and actions as she tries to achieve her goal with the software. It’s amazing how many potholes you can discover with that simple process.

Fundamentally, the implicit web is about magic. It’s about weaving disorganized information into something useful. The danger comes because you’re filling in the blanks for the user, guessing at what information they want to see rather than relying on them asking for something explicit. The user pays a price every time they have to process the information you give them. If you throw irrelevant data at them, they’ll kick you to the curb, to join Clippy in his retirement.

Funhouse Photo User Count: 1638 total, 77 active. A friend just pointed out that it’s unclear if your profile picture will get replaced when you pick from the app for the first time. That seems obvious now he says it, so I’ll be looking at making that clearer. A good example of the sort of thing it’s easy to miss when you’re focused on the technical problems.

Event Connector User Count: 46 total, 8 active. Still working on getting an ‘anchor’ conference for the app.

The implicit web is all about analyzing information a user has generated as part of some activity, and giving them a new way of looking at that data, generating insights they wouldn’t otherwise see. The most common source of information is web browsing, which sites are visited, which links are clicked. There is a source that’s just as interesting, but nobody’s using it; your mailbox.

Who sends mail to you, and who you respond to, how often and at what length. If you take that raw data and plot it in a graph, with links between you and people you correspond with, you end up with a pretty damn accurate graph of your relationships, limited only by the extent to which you use email to interact with your friends and colleagues. What’s more, by using the frequency of correspondence, you can approximate the strength of each relationship, and by seeing who else is included in emails, have a sketchy idea of the links between your friends.

As well as a social graph, you can also look through mail for links to web pages. Each one of those can be treated just as if that friend had voted for it in a service like Digg, and can be added to a list of the sites recommended by your local network.

Why’s no one doing this, if it’s so wonderful? Because it’s really, really hard to get to that data. Web-based email services like Google Mail or Hotmail would be able to do it on the server side, but that approach requires a large existing user-base. An alternative I’m investigating is writing an Outlook Add-In to gather some of this information. This would restrict me to business users, and involves wrestling the COM beast to the ground to implement, but I should be able to reuse some of my Internet Explorer BHO work at least.

Funhouse Photo User Count: 1,552 total, 123 active. This is encouraging, another peak in the growth rate, and no public holiday to account for it this time. Perhaps the graph is becoming slightly less linear?

Event Connector User Count: 41 total, 10 active. Not much change. I’m still talking with promoters, trying to arrange official support for another conference.