Accessing mail with Extended MAPI

Stamp
In the first part of this series, I gave an overview of the APIs you can use to access Outlook mail, along with directions for how to get started with Visual Basic for Applications.

As I mentioned, using the Outlook Object Model/VBA is both slow and throws up a lot of security warnings. I need to process a lot of mail messages, and I need a language that will handle both heavy data processing and advanced graphics for the work I’ll need to do to turn the messages into a graph. This means I need a non-interpreted language with a lot of libraries, such as Java, C# or C++. Luckily, Extended MAPI is an old-school COM API that fits well with C++, and avoids the security restrictions.

One thing I didn’t realize about MAPI is that it’s usable even if you’re not running as an Outlook plugin. It’s designed to let third-party programs interact with the user’s mail store, and is available as long as Outlook has been installed on the system. This OutlookCode.com page is the most in-depth description and bibliography I’ve found.

The down-side of MAPI is that it’s COM, offers a lot of functionality, and has been implemented slightly differently by each version of Office. This means that using it can be tough, just getting to the ‘hello world’ level requires a lot of work, and past that there’s a lot of pits with spikes at the bottom to wander into. What I needed was a good code example to get me started.

Lucian Wischik came to the rescue, with some wonderful sample code demonstrating exactly how to use MAPI, along with some utility functions to simplify access, ample commentary explaining what was going on, and a completely public domain license for use. Amazingly, this is the third time in recent months that I’ve found what I was looking for on Lucian’s site. He also has the best samples on the web for writing a BHO without ATL, and hosting IE web rendering inside your own window. He must be accruing some serious good karma.

With Lucian’s help, it’s not too hard to create your own standalone executable that enumerates Outlook folders and items, and extracts information from them. Unfortunately, it’s not always possible to pull out the full HTML body of the message, but I’m mostly interested in the sender and receivers of each email which is easily accessible. It runs a lot faster than the equivalent VBA code, and doesn’t bring up any security dialogs.

Funhouse Photo User Count: 1,892 total, 101 active. Almost exactly the same as last night, so I must be looking at the same sample window in the stats.

Event Connector User Count: 82 total, 4 active. Also very similar.

So L.A.

Californiaevening

I just talked to my neighbor, who’s having his back wall rebuilt after a car flew off the freeway into his pool. Liz spent an hour cleaning the ash out of our hot-tub. Dinner was an In-N-Out animal-style burger, but I’ll be working that off with a ride along the beach path in Santa Monica, after a 90 minute commute on the 405. This weekend, fires permitting, we’ll be camping with the coyotes, in the mountains north of Malibu.

I don’t stop to think about it as often as I should, but when I do it always blows my mind. As a kid in Britain I grew up watching this world on TV and dreaming. Now I’m living in it! There’s so many things and people I love and miss in the UK, but this feels like home.

Funhouse Photo User Count: 1,892 total, 109 active. It’s been good to see the active count increasing along with the total. I need to graph this again.

Event Connector User Count: 82 total, 5 active. Still picking up some random users from the directory, but no takers from event organizers.

Outlook plugin basics

Postcard

My current mission to create a social graph from my mailbox. I need to get access to the raw data on my email habits, and the easiest platform for that is Outlook, since it offers a plugin API. I had no clue how to get started with it though, and most articles I could find assumed some level of familiarity with the technology, so here’s a guide covering what I discovered on the absolute basics.

The first thing I discovered is, like standards, that the great thing about Outlook APIs is that there’s so many to choose from! The second discovery was that the flood of email viruses using those APIs a few years ago led Microsoft to barricade most of them behind heavy security. Luckily, there’s a lot of people writing Outlook code, and so there’s a lot of examples and resources to learn from. The best starting point is http://outlookcode.com/, but plain google searching will usually throw up some answers to even fairly obscure questions.

Collaboration Data Objects (CDO)
are a legacy interface API, and using them will often bring up scary security dialogs, since Outlook assumes you’re doing something malicious.

The Outlook Object Model (OOM) is the most common way of working with the data held by Outlook. It the interface you use by default when writing Visual Basic for Applications macros within the app, it’s well documented and supported, though you have to be careful for differences between app versions. Unfortunately for us, it’s also hedged in by security warning dialogs and restrictions. Because you generally have to make an API call for each mail item you want to deal with, it can be very slow if you want to handle thousands of them at once.

Extended MAPI is an advanced interface to the mail store in Outlook, and it’s mostly commonly used from from C++ or other compiled languages. The good news is that you’ve got access to almost any information, with no security UI. The bad news is that it’s not well documented, and is tricky to work with. Because you can access lots of items at once, without the overhead of an interpreted call every time, it can be a lot faster when you’re working with large numbers of emails.

Redemption is a third-party library that combines a lot of the advantages of Extended MAPI and OOM. Under the hood it uses MAPI, and so it’s fast and avoids security warnings, but it exposes an interface that’s a carbon copy of OOM, which makes it simple to use. It’s a real feat of engineering by Dmitry Streblechenko, and he’s also been very prolific in answering Outlook programming questions. At only $200 for an unlimited distribution version, it’s a great deal.

Getting started prototyping your code is as simple as pressing Alt+F11 in Outlook. This will bring up a separate VBA code window. If you select the session in the left pane, then take any of the simple examples out there and paste them in, you should be able to run them by hitting F5.

To make Redemption available, run its installer and then it should appear in the code editor’s Tools->References dialog box as an option. Then just reference its objects like any of the built-in types.

I’ve managed to create a simple prototype that loops through all the messages in a folder and outputs the basic information I need to build my graph to a text file. It’s slow, probably only a few dozen messages a second, but it should be good enough to let me create the foundation for a social graph. I’m moving over several years worth of email now, so I’ll have a good data-set to work with.

Funhouse Photo User Count
: 1,847 total, 93 active. I had a new review, requesting larger photos. That’s an interesting idea, if more people had photos in their albums that could be a compelling feature. I also saw the stats showing more people removing than adding the app, which doesn’t seem to match the total figures!

Event Connector User Count
: 77 total, 4 active. Still very quiet.

Games, UI, and the implicit web

Joypad
I was a console programmer for six years. Games are the only pieces of software that people use purely for the joy of interacting with a computer. There’s no reason to play, except to have fun.

This means that the user interface is crucial. With other software, people will put up with the pain of a bad UI because they’re trying to accomplish some real-world task. If a consumer picks up a video game and it doesn’t let them have fun within a minute or two, they will give up on it. The interface has to be easy and fun. It can still be deep, but that complexity must be intuitive and discoverable, and not presented like the Space Shuttle’s control panel.

What really excites me about the implicit web is the promise of using the gathered data to turbo-charge everyday interfaces. A simple example is Firefox’s address bar; it remembers the URLs I visit, and when I start typing a new one, the suggestions are in most-visited order. By contrast, I wouldn’t class Google Suggest in the search box as an implicit service, since it doesn’t customize the suggestions based on my behavior, and it’s a lot less useful for me.

When I was working with Nintendo, the holy grail was the ‘one button game’. Think Mario 64, where you managed complex interactions with a 3D world mostly with the joystick and a single button to jump. Stumbleupon is the web service that’s closest to this, I’ve heard it described as the ‘Forward button‘ for the web, and it really delivers a lot of value with very little input needed from the user. Google Hot Keys is my attempt to move searching in that direction, though there’s no implicit component.

One of the parts I’m most anticipating about Defrag is seeing all of the innovative interfaces that the teams will be showing off. There’s so many possibilities for improving the user experience, I can’t wait to see what people are coming up with!

Funhouse Photo User Count: 1,817 total, 93 active. Still growing steadily, but slowly, with most of the additions coming through the product directory.

Event Connector User Count: 77 total, 4 active. Still working with conference organizers, not much to show yet though.

Hiking in the Angeles – Condor Peak Trail

Angelesmountains

Yesterday me and Liz made it up to the Angeles mountains just north of Pasadena. Rising up steeply from the valley, the peaks reach around 6000 feet, giving both great views and some punishing climbs. Liz had just bought a Tom Harrison map of the western end of the Angeles, and we picked a promising trail, Condor Peak. I’ve put up a Google map covering the route we took, it turned out to be a lot of fun.

The drive up there was pretty short. We got off the 210 at Sunland, then onto Oro Vista and continued around 8 miles as it turned into Big Tujunga Canyon Road. The road was heavily potholed at the start, but got better, and wasn’t too twisty which made for a more pleasant drive. There was only one other car in the parking lot, and we met them a short way up the trail coming back after they’d only gone a little way.

The trail started off by switchbacking up from the lot, but then descended and headed back toward the road. At about a mile and a half, there’s a side-trail that forks off and heads down to a picnic area near the road, which might be an option as a starting point if you want to skip the first section. One nice feature of the trail is the provision of mile markers. They start off with roman numerals until the fourth mile, which has an arabic ‘4’, and then after that they give up marking the numbers. They’ve done a great job building the trail along some very steep rocky slopes, and they keep a tough but steady grade for most of way. Given the terrain, that’s quite a feat of route planning. It worked out to around 500 feet up every mile, which is work but hike-able for us.

There’s a lot of switchbacking up and along ridges, passing through scrubby chapparal with some of the biggest Yucca I’ve ever seen. After about the fourth mile, you start to get some views over the reservoir, and across to Josephine Peak and into the Valley. We started late, so we made it about six miles up before we had to turn around, at around 4300′ elevation, 2000 feet higher than the trail head. The trail was visible and clear ahead of us, but I’ve only mapped it as far as we walked. It looked like you’d get some really good vistas if you continued up to the eight mile point, and Condor peak at 5070′.

View Larger Map

Funhouse Photo User Count: 1,777 total, 75 active. I ended up working on my day job most of today, along with some halloween decorating, so I didn’t get to make any of the changes I was planning.

Event Connector User Count: 77 total, 7 active. Not much change, but I did run across the Web Community Forum in my research. They look like a very clued-up team focused on using social tools to build communities online, and they already have a high level of Facebook integration on their blog, with links to themselves, their group, and an event for their upcoming conference.

Wildfire coming our way

Smokeysky

That’s the view from our house, those clouds are all smoke. There’s a large wildfire a few miles upwind of us, and we’re getting the full show, red sun, falling ash and all! We’re in little danger here since they built a mall on the grassland across from our house, but I hope that the fire-fighters we know like Frank Padilla and Dave Updike are taking care.

There’s two major local fires. With winds gusting up to 90 mph for the next three days, and a drought that’s left the brush dry as tinder, they could both cover a lot of ground before they’re controlled. Down in Malibu, on the south side of the Santa Monica Mountains, the fire’s smaller but in a heavily settled area and has already taken out some houses. The Castaic fire’s to the north of us, it’s much bigger but is burning through wild back country. There’s also a lot of smaller blazes as the wind knocks down power lines.

We know from our trail-building experience how much fuel has built up in the Santa Monicas. In some places, there’s dead brush above head-height, it’s been decades since a fire’s been through. I’m hoping there’s been enough warning for everyone to get out of harm’s way, because those sections will burn hot and fast.