Author Archives: Pete Warden

Implicit web philately

October 26, 2007 By in Defrag, Implicit Web Leave a comment

Stamps

With Defrag fast approaching, I’ve been spending some cycles thinking about what the Implicit Web actually is, and where it’s going. When I’m staring at this sort of problem, a technique I find really useful is "stamp collecting". Gather as many examples as possible, list their important properties, group them into clusters and look at what patterns emerge.

Here’s my current list of Implicit Web services currently out there, with a couple that are on the borderline of the term. I’ve not got enough to meaningfully group them, so they’re alphabetical:

Adaptive Blue – More of a semantic web application, but they do offer a Firefox extension. Being client-based is a distinguishing feature of implicit web apps, since that’s the only way I know to get access to the user data needed.

Amazon – Their recommendation system is the grand-daddy of a lot of the apps that take raw information on user’s behavior, run some magic algorithms, and return something useful back to the customer. It’s a hard trick for most startups to repeat, since almost nobody has access to the Amazon’s breadth of data. This is why client-based solutions that can track behavior across many sites seem like the only practical solution.

last.fm – A true implicit web app, they have client-based tracking of the user’s behavior, they piggy-back on other people’s applications to gather their data and use that to return Amazon-style recommendations. It does make me wonder about the ‘web’ part of the term though, since that seems to imply web browsing. Maybe ‘implicit internet’ would be more appropriate?

me.dium – Another app that fully fits the term. A unique feature is that they use the social graph to combine information from multiple users, which I think is a very promising area for implicit web applications. Being able to pool data from your friends is a great way of discovering relevant new content.

MySportsNet.ca – This is one I came across relatively recently. It’s a client-side app that monitors your browsing, and tailors a sports portal site to match your interests based on that data. What’s really interesting is that it’s aimed at a mainstream audience of sports fans, rather than geeky early adopters. I know from my game career that the sports audience is massive, and willing to pay for something ties into their passion, so I’ll be following its progress closely. The only audience I know that’s similar is music, and it’s relevant that the most successful implicit app so far, last.fm, tapped into that demand.

tape failure – This is a service I’ve only read about, but unfortunately their site seems to be down at the moment. They’re not an implicit web app at all, but it does seem like they have a good solution to the browsing data collection problem.

Let me know if you think I’m missing any. I may put together a page tracking new services, since I think we’re going to be getting a lot more over the next year.

Funhouse Photo User Count: 1,916 total, 92 active. The proportion of profile-box adds was a bit higher this time, which is promising because it scales a bit more virally than the product directory.

Event Connector User Count: 84 total, 9 active. Not much happening on this front.

Accessing mail with Extended MAPI

October 25, 2007 By in Outlook API 2 Comments

Stamp
In the first part of this series, I gave an overview of the APIs you can use to access Outlook mail, along with directions for how to get started with Visual Basic for Applications.

As I mentioned, using the Outlook Object Model/VBA is both slow and throws up a lot of security warnings. I need to process a lot of mail messages, and I need a language that will handle both heavy data processing and advanced graphics for the work I’ll need to do to turn the messages into a graph. This means I need a non-interpreted language with a lot of libraries, such as Java, C# or C++. Luckily, Extended MAPI is an old-school COM API that fits well with C++, and avoids the security restrictions.

One thing I didn’t realize about MAPI is that it’s usable even if you’re not running as an Outlook plugin. It’s designed to let third-party programs interact with the user’s mail store, and is available as long as Outlook has been installed on the system. This OutlookCode.com page is the most in-depth description and bibliography I’ve found.

The down-side of MAPI is that it’s COM, offers a lot of functionality, and has been implemented slightly differently by each version of Office. This means that using it can be tough, just getting to the ‘hello world’ level requires a lot of work, and past that there’s a lot of pits with spikes at the bottom to wander into. What I needed was a good code example to get me started.

Lucian Wischik came to the rescue, with some wonderful sample code demonstrating exactly how to use MAPI, along with some utility functions to simplify access, ample commentary explaining what was going on, and a completely public domain license for use. Amazingly, this is the third time in recent months that I’ve found what I was looking for on Lucian’s site. He also has the best samples on the web for writing a BHO without ATL, and hosting IE web rendering inside your own window. He must be accruing some serious good karma.

With Lucian’s help, it’s not too hard to create your own standalone executable that enumerates Outlook folders and items, and extracts information from them. Unfortunately, it’s not always possible to pull out the full HTML body of the message, but I’m mostly interested in the sender and receivers of each email which is easily accessible. It runs a lot faster than the equivalent VBA code, and doesn’t bring up any security dialogs.

Funhouse Photo User Count: 1,892 total, 101 active. Almost exactly the same as last night, so I must be looking at the same sample window in the stats.

Event Connector User Count: 82 total, 4 active. Also very similar.

So L.A.

October 25, 2007 By in Personal Leave a comment

Californiaevening

I just talked to my neighbor, who’s having his back wall rebuilt after a car flew off the freeway into his pool. Liz spent an hour cleaning the ash out of our hot-tub. Dinner was an In-N-Out animal-style burger, but I’ll be working that off with a ride along the beach path in Santa Monica, after a 90 minute commute on the 405. This weekend, fires permitting, we’ll be camping with the coyotes, in the mountains north of Malibu.

I don’t stop to think about it as often as I should, but when I do it always blows my mind. As a kid in Britain I grew up watching this world on TV and dreaming. Now I’m living in it! There’s so many things and people I love and miss in the UK, but this feels like home.

Funhouse Photo User Count: 1,892 total, 109 active. It’s been good to see the active count increasing along with the total. I need to graph this again.

Event Connector User Count: 82 total, 5 active. Still picking up some random users from the directory, but no takers from event organizers.

Outlook plugin basics

October 23, 2007 By in Outlook API Leave a comment

Postcard

My current mission to create a social graph from my mailbox. I need to get access to the raw data on my email habits, and the easiest platform for that is Outlook, since it offers a plugin API. I had no clue how to get started with it though, and most articles I could find assumed some level of familiarity with the technology, so here’s a guide covering what I discovered on the absolute basics.

The first thing I discovered is, like standards, that the great thing about Outlook APIs is that there’s so many to choose from! The second discovery was that the flood of email viruses using those APIs a few years ago led Microsoft to barricade most of them behind heavy security. Luckily, there’s a lot of people writing Outlook code, and so there’s a lot of examples and resources to learn from. The best starting point is http://outlookcode.com/, but plain google searching will usually throw up some answers to even fairly obscure questions.

Collaboration Data Objects (CDO) are a legacy interface API, and using them will often bring up scary security dialogs, since Outlook assumes you’re doing something malicious.

The Outlook Object Model (OOM) is the most common way of working with the data held by Outlook. It the interface you use by default when writing Visual Basic for Applications macros within the app, it’s well documented and supported, though you have to be careful for differences between app versions. Unfortunately for us, it’s also hedged in by security warning dialogs and restrictions. Because you generally have to make an API call for each mail item you want to deal with, it can be very slow if you want to handle thousands of them at once.

Extended MAPI is an advanced interface to the mail store in Outlook, and it’s mostly commonly used from from C++ or other compiled languages. The good news is that you’ve got access to almost any information, with no security UI. The bad news is that it’s not well documented, and is tricky to work with. Because you can access lots of items at once, without the overhead of an interpreted call every time, it can be a lot faster when you’re working with large numbers of emails.

Redemption is a third-party library that combines a lot of the advantages of Extended MAPI and OOM. Under the hood it uses MAPI, and so it’s fast and avoids security warnings, but it exposes an interface that’s a carbon copy of OOM, which makes it simple to use. It’s a real feat of engineering by Dmitry Streblechenko, and he’s also been very prolific in answering Outlook programming questions. At only $200 for an unlimited distribution version, it’s a great deal.

Getting started prototyping your code is as simple as pressing Alt+F11 in Outlook. This will bring up a separate VBA code window. If you select the session in the left pane, then take any of the simple examples out there and paste them in, you should be able to run them by hitting F5.

To make Redemption available, run its installer and then it should appear in the code editor’s Tools->References dialog box as an option. Then just reference its objects like any of the built-in types.

I’ve managed to create a simple prototype that loops through all the messages in a folder and outputs the basic information I need to build my graph to a text file. It’s slow, probably only a few dozen messages a second, but it should be good enough to let me create the foundation for a social graph. I’m moving over several years worth of email now, so I’ll have a good data-set to work with.

Funhouse Photo User Count: 1,847 total, 93 active. I had a new review, requesting larger photos. That’s an interesting idea, if more people had photos in their albums that could be a compelling feature. I also saw the stats showing more people removing than adding the app, which doesn’t seem to match the total figures!

Event Connector User Count: 77 total, 4 active. Still very quiet.

Games, UI, and the implicit web

October 22, 2007 By in Defrag, Implicit Web Leave a comment

Joypad
I was a console programmer for six years. Games are the only pieces of software that people use purely for the joy of interacting with a computer. There’s no reason to play, except to have fun.

This means that the user interface is crucial. With other software, people will put up with the pain of a bad UI because they’re trying to accomplish some real-world task. If a consumer picks up a video game and it doesn’t let them have fun within a minute or two, they will give up on it. The interface has to be easy and fun. It can still be deep, but that complexity must be intuitive and discoverable, and not presented like the Space Shuttle’s control panel.

What really excites me about the implicit web is the promise of using the gathered data to turbo-charge everyday interfaces. A simple example is Firefox’s address bar; it remembers the URLs I visit, and when I start typing a new one, the suggestions are in most-visited order. By contrast, I wouldn’t class Google Suggest in the search box as an implicit service, since it doesn’t customize the suggestions based on my behavior, and it’s a lot less useful for me.

When I was working with Nintendo, the holy grail was the ‘one button game’. Think Mario 64, where you managed complex interactions with a 3D world mostly with the joystick and a single button to jump. Stumbleupon is the web service that’s closest to this, I’ve heard it described as the ‘Forward button‘ for the web, and it really delivers a lot of value with very little input needed from the user. Google Hot Keys is my attempt to move searching in that direction, though there’s no implicit component.

One of the parts I’m most anticipating about Defrag is seeing all of the innovative interfaces that the teams will be showing off. There’s so many possibilities for improving the user experience, I can’t wait to see what people are coming up with!

Funhouse Photo User Count: 1,817 total, 93 active. Still growing steadily, but slowly, with most of the additions coming through the product directory.

Event Connector User Count: 77 total, 4 active. Still working with conference organizers, not much to show yet though.

Hiking in the Angeles – Condor Peak Trail

October 21, 2007 By in Personal Leave a comment

Angelesmountains

Yesterday me and Liz made it up to the Angeles mountains just north of Pasadena. Rising up steeply from the valley, the peaks reach around 6000 feet, giving both great views and some punishing climbs. Liz had just bought a Tom Harrison map of the western end of the Angeles, and we picked a promising trail, Condor Peak. I’ve put up a Google map covering the route we took, it turned out to be a lot of fun.

The drive up there was pretty short. We got off the 210 at Sunland, then onto Oro Vista and continued around 8 miles as it turned into Big Tujunga Canyon Road. The road was heavily potholed at the start, but got better, and wasn’t too twisty which made for a more pleasant drive. There was only one other car in the parking lot, and we met them a short way up the trail coming back after they’d only gone a little way.

The trail started off by switchbacking up from the lot, but then descended and headed back toward the road. At about a mile and a half, there’s a side-trail that forks off and heads down to a picnic area near the road, which might be an option as a starting point if you want to skip the first section. One nice feature of the trail is the provision of mile markers. They start off with roman numerals until the fourth mile, which has an arabic ‘4’, and then after that they give up marking the numbers. They’ve done a great job building the trail along some very steep rocky slopes, and they keep a tough but steady grade for most of way. Given the terrain, that’s quite a feat of route planning. It worked out to around 500 feet up every mile, which is work but hike-able for us.

There’s a lot of switchbacking up and along ridges, passing through scrubby chapparal with some of the biggest Yucca I’ve ever seen. After about the fourth mile, you start to get some views over the reservoir, and across to Josephine Peak and into the Valley. We started late, so we made it about six miles up before we had to turn around, at around 4300′ elevation, 2000 feet higher than the trail head. The trail was visible and clear ahead of us, but I’ve only mapped it as far as we walked. It looked like you’d get some really good vistas if you continued up to the eight mile point, and Condor peak at 5070′.

View Larger Map

Funhouse Photo User Count: 1,777 total, 75 active. I ended up working on my day job most of today, along with some halloween decorating, so I didn’t get to make any of the changes I was planning.

Event Connector User Count: 77 total, 7 active. Not much change, but I did run across the Web Community Forum in my research. They look like a very clued-up team focused on using social tools to build communities online, and they already have a high level of Facebook integration on their blog, with links to themselves, their group, and an event for their upcoming conference.

Wildfire coming our way

October 21, 2007 By in Personal Leave a comment

Smokeysky

That’s the view from our house, those clouds are all smoke. There’s a large wildfire a few miles upwind of us, and we’re getting the full show, red sun, falling ash and all! We’re in little danger here since they built a mall on the grassland across from our house, but I hope that the fire-fighters we know like Frank Padilla and Dave Updike are taking care.

There’s two major local fires. With winds gusting up to 90 mph for the next three days, and a drought that’s left the brush dry as tinder, they could both cover a lot of ground before they’re controlled. Down in Malibu, on the south side of the Santa Monica Mountains, the fire’s smaller but in a heavily settled area and has already taken out some houses. The Castaic fire’s to the north of us, it’s much bigger but is burning through wild back country. There’s also a lot of smaller blazes as the wind knocks down power lines.

We know from our trail-building experience how much fuel has built up in the Santa Monicas. In some places, there’s dead brush above head-height, it’s been decades since a fire’s been through. I’m hoping there’s been enough warning for everyone to get out of harm’s way, because those sections will burn hot and fast.

Slinky companies and public transport

October 19, 2007 By in Addon Promotion, Defrag, Facebook API, GoogleHotKeys, Implicit Web Leave a comment

Slinky
Yesterday, Brad posted an article talking about bubble times in Boulder, and quoted a great line from Bill Perry about how they spawned ‘slinky companies’ that "aren’t very useful but they are fun to watch as they tumble down the stairs".

Rick Segal had a post about why he took the train to work, and how people-watching there was a great reality check to a lot of the grand technology ideas he was presented with.

And via Execupundit, I came across a column discussing whether people were really dissatisfied with their jobs, or just liked to gripe and fantasize. One employee who’d been involved in two start-ups that didn’t take off said "Most dreams aren’t market researched."

These all seemed to speak to the tough balance between keeping your feet on the ground and your eyes on the stars. As Tom Evlin’s tagline goes, "Nothing great has ever been accomplished without irrational exuberance." I’ve been wrestling with how to avoid creating a slinky with technology that sounds neat enough to be funded, but will never amount to anything. To do that, I’ve focused on solving a painful problem, and validating both the widespread existence of the problem, and that people like my solution.

I’ve turned my ideas into concrete services, and got them into the wild as quickly as possible. Google Hot Keys has proved that it’s possible to robustly extract data from screen-scraping within both Firefox and IE, but its slow take-up suggests there isn’t a massive demand for a swankier search interface. Defrag Connector shows that being able to connect with friends before a conference is really popular, but the lack of interest so far in Event Connector from conference promoters I’ve contacted shows me it won’t just sell itself. Funhouse Photo’s lack of viral growth tells me that I need to provide a compelling reason for people to contact their friends about the app, and not just rely on offering them tools to do so.

I really believe in all of these projects, but I want to know how to take them forward by testing them against the real world. All my career, I’ve avoided grand projects that take years before they show results. I’ve been lucky enough that all of the dozen or so major applications I’ve worked on have shipped, none were cancelled. Part of that is down to my choice of working on services that have tangible benefits to users, and can be prototyped and iteratively tested against that user need from an early stage. Whether it’s formal market research, watching people on trains, or just releasing an early version and seeing what happens, you have to test against reality.

I’m happy to take the risk of failing, there’s a lot of factors I can’t control. What I can control is the risk of creating something useless!

Funhouse Photo User Count: 1,746 total, 70 active. Much the same as before, I haven’t made any changes yet.

Event Connector User Count: 73 total, 9 active. Still no conference takeup. I did experiment with a post to PodCamp Boston’s forum to see if I could reach guests directly, but I think the only way to get good distribution is through the organizers.

Facebook and event promotion

October 18, 2007 By in Defrag, Facebook API Leave a comment

Partyhat
As I’ve been approaching conference organizers to try Event Connector, I’ve been surprised at how few have Facebook events. It seems like a no-brainer to me if your audience includes anyone under thirty, since it only takes a couple of minutes to create an event. In return, you get a great platform for potential guests to discover your conference, and attendees to hear from you and each other before and after the event. You’re being given permission to market to them, and even better, the participants themselves will spread the word as their attendance shows up on their friends’ feeds, and they get involved on the discussions on the event page itself.

Most of the events I have run across have been unofficial, started by participants rather than organizers. Without publicity from the promoters, these tend to attract only a few guests. To be effective you need to include a link to the event in some material that goes out to a decent number of your guests.

I don’t think it’s that conference organizers don’t want the benefits that facebook events offer, since I see a lot of organizations trying to hand-roll similar services. PodCamp Boston has a page listing all of the attendees who wanted their names to be public, but as a plain text alphabetical list, it’s a lot harder to discover friends than the equivalent on facebook. Facebook events are popular with guests, the New Media Expo 2008 one picked up over a hundred guests in the first few hours after it was created, and this is for an event almost a year away!

Trying to put myself in their shoes, I’d guess that the main obstacles are the fact that no one else is doing it, it’s an unknown quantity, it feels a bit out of their control, and they’ve never needed it before. It does require a willingness to try something new, but the reward for doing so before it’s mainstream is that you’ll get a lot of buzz, publicity and guest goodwill for taking that leap!

If you’re an event promoter, I’d highly recommend you set up a Facebook event, and give it a little promotion. It’s quick, free, and offers both you and your guests significant benefits.

Even better, once you’ve got one set up, you get an Event Connector for free. Go to the main page of the app, and your event will show up at the top. There’s a link you can mail out, and free blogger, typepad and facebook profile badges you can distribute. It adds value to the plain facebook events by allowing users to see which of their friends, and friends-of-friends, are going, which supplies the social proof that will persuade them to sign up.

Funhouse Photo User Count: 1,729 total, 63 active. The same steady growth, and looking at the breakdown, I see the same pattern of non-viral acquisition of users, mostly through the directory and searches.

Event Connector User Count: 72 total, 8 active. Still very quiet, with no conference signed up, and a trickle of users from the directory.