Beetlejuice, Beetlejuice, Beetlejuice


When I was 16, I’d sneak into a University of Cambridge computer lab, with the desire to download Amiga mods and demos from ftp sites. This was pre-web, but I owned The New Hacker’s Dictionary (aka the Jargon File), and had been very excited by the concepts of email and Usenet. What blew me away when I actually tried them was that the person who wrote my assembler reference was on the ARM newsgroup and answering questions from mere mortals like me!

I was reminded of that yesterday when Matt Brezina from Xobni left a comment here after I reviewed their product. I’m old enough to remember a time before the internet, when people just couldn’t connect like that. Now there’s services that let you know when someone talks about something you’re tracking, and start a conversation. That’s what I find really interesting about working with a social graph, building tools that help people build relationships.

That computer lab also led to me living in an Alaskan treehouse for three months, but that’s a story for another post!

Funhouse Photo User Count: 2,003 total, 73 active. Broke 2000!

Event Connector User Count: 100 total, 5 active. Broke 100, which isn’t quite as exciting.

Competing mail graph services

My idea of deriving a social graph from your email messages isn’t new. In a simple form, sites like Facebook already pull your contacts from webmail sites to build a first draft of your friends list. Unfortunately, your contacts list is a poor map of your actual relationships.

SNARF is a Microsoft project from 2005 which analyzes your email in a much more sophisticated way, and uses this information to help you triage your messages. It has a flexible system of metrics, for example the number of emails you’d sent someone, to calculate an importance score for each contact. Emails from important people are displayed in a top priority area of the UI, away from the less important bacn.

Xobni is a more recent Outlook add-in that analyzes your messages. The slick UI gives you a lot of interesting ways to drill down into the relationships it finds. It gives you fast searching, raw stats on each contact’s mailing patterns, and lots more. Its automatic phone number extraction from messages and the display of attachments by sender look particularly useful.

Looking further afield, there’s some overlap with spam protection services. Spam guards attempt to exclude a particular class of email by analyzing messages. I don’t know of any that go beyond checking if you’ve sent someone an email before in their social analysis, but it seems like a natural direction for some of those companies to head in.

Yahoo and Google both have access to the raw information from millions of mail users, so if they see an advantage in this sort of mail-based social graph, they could create something really compelling. One interesting area is the APIs they offer, which might be enough for a third-party developer to at least show a proof-of-concept demo, though it would no doubt be against the ToS.

What I’ve yet to see if a really painful problem that is solved by any of these services. Xobni is the closest, but it still feels like a great set of additional features for Outlook, not a make-the-pain-stop solution that people will pay for.

I’m certain there are some great ways to solve problems using a mail-based social graph, I just need to find them!

Funhouse Photo User Count: 1,998 total, 73 active. Almost up to 2000 total,  with no work on it for the last month.

Event Connector User Count: 91 total, 5 active. A few new users still coming in through the directory, but no conference sign-ups.

Take a walk on the client-side


I’ve been following Mozilla’s Prism launch with a lot of interest. One comment that caught my eye was from  Ryan Stewart, who believes "that the desktop isn’t dead, and that a hybrid approach is a successful way to go". There’s a lot of opportunities for really useful services that need a client-side component, and that’s key to my strategy. It’s the reason I’ve worked on Java Applets, Firefox and Internet Explorer extensions and now an Outlook plugin.

Web-based apps have a lot of advantages over traditional desktop apps:
No install needed! It’s an incredible advantage to be able to go to a URL and instantly start using the service. The installation process is a big barrier to people even trying your app, since they’ve been trained to know it takes minutes, involves answering questions they may not know the answer to, and worrying about their configuration.
Complete safety. There’s no worries about viruses or spyware, the only information it has access to is what you type in.
Use anywhere. You can use the app from any machine in the world, regardless of OS or configuration.
Up-to-date and comprehensive information. Since the information you’re accessing is pulled live from a server, it can be updated instantly, along with the application itself.
Easy to develop. You have a lot fewer variables to worry about with a server-based app. You know exactly what hardware and software the main code is running on. Keeping the output HTML cross-platform is a couple of orders-of-magnitude easier than doing the same for executable code.

There are some obstacles that I think will prevent pure web-based services from taking over the app world:
A limiting sandbox. To achieve security, web-pages can only pull data from their own servers, and can’t use any of the information already on the user’s machine, or any other web-services the user is signed onto, without an explicit agreement with the provider. This effectively stovepipes all information, and is the reason I’ve entered my list of friends on at least a dozen different services. I don’t see this changing, because it would require a majority of the browser vendors to implement a more subtle security policy than the current blanket same-domain-policy. Loosening security like that doesn’t seem likely.
Poor UI. Google has done an astonishing job with their mail interface, but for power users the experience still doesn’t match Outlook or Apple Mail. Running within a web-browser makes it tough to offer a rich interface, and you can’t use the standard OS metaphors. This is where Prism is interesting, because XUL actually offers some of the UI that HTML is missing, like sliders.

What I think is really interesting is the idea of combining the strengths of the two approaches. To start off, there needs to be a pure web interface to build up enough interest and trust for people to download a client extension that offers more features. That extension can then offer some tools to the app, like cross-domain access and native UI, but still keeping the bulk of the application running on the server. That keeps the big advantages of SAAS, such as ease of development, and online access to information, but allows a whole range of applications that aren’t possible now.

Funhouse Photo User Count
: 1,975 total, 77 active.

Event Connector User Count: 89 total, 1 active.

La Jolla Valley hike-in campground


Liz and I just got back from an overnight backpacking trip in the local mountains. We had a new tent and other gear we wanted to test-drive, and didn’t want to be too stuck if some of it didn’t work. With this in mind, we chose the La Jolla Valley campground, since the trailhead’s less than an hour from our house, and it’s only a 2.5 mile hike in.

As I mentioned in a previous post, this campground is the only one I know of in the Santa Monicas that doesn’t require reservations, and it’s rare to see anybody using it, so we felt pretty confident we would get a spot. The recent wildfires to the east probably kept people away too. We’d checked the weather and the red-flag status before-hand, along with asking a friendly wildland firefighter friend, and it looked like there was no danger.

We stopped by the booth at Thornhill Broome Beach when we arrived, paid $3 per-person to camp and picked up a parking pass. We then drove across the street to the day-use parking lot, where the trail starts. There’s two trails leading off the lot, the Ray Miller section of the Backbone, and La Jolla Canyon trail. To get to the campground, we had to hike 2.5 miles up the La Jolla trail, which is the one behind the yellow gate, that starts off like a fire-road.

You start off with a moderate uphill grade going along the canyon floor. After about half a mile you cross over the wash, and start heading towards a waterfall. Sadly, with the drought, there was no water left when we got there, but it’s still a nice spot to catch your breath. The trail heads up the side of the waterfall, and then starts switchbacking up the canyon wall. The next half-mile is probably the toughest part of the hike, with poor footing and some steep grades.

At around 1.2 miles, you’ll reach a fork in the trail. Keep to the right/eastern side at this fork to get to the campground. The left/western trail heads towards Mugu peak, and connects with some of the other trails criss-crossing the valley to form some fun loops.

The trail is still uphill, but with a decent grade, and goes through some nice shady brush archways. 1.8 miles in, you get to another fork, this one with a new sign:

It’s hard to tell from the sign, but you actually want to head left/west for the quickest route to the campground. It’s a short connector that skirts a pond, and the first camping spots are just off to the west of it. We didn’t investigate these, we knew there were some more further on, and the way to them looked a bit rough. The trail reaches a fire-road, and there’s camp spots just off this road, a short distance west. There’s no map of the camping spots, so you’ll just need to look for the numbered signs and explore.

We went to site 5, which was secluded, in a small oak grove. There’s no camp-fires or water, but they do have a pit-toilet nearby, and gas-stoves are allowed. Slightly further west along the fire-road is a group camp site, that looks like it could handle up to ten tents comfortably. We did notice the restroom had been knocked off its foundation by a fallen tree, so you’ll need to use the toilet that’s near the individual sites instead.

It was a pretty gruelling hike, we’d brought too much water, and there’s about 600 feet of elevation gain, so we were very happy to sit down at the picnic table and rest our feet. It was actually raining for once, so we got our new tent up as quickly as we could, to keep our stuff dry. I’m very happy with the REI Half Dome tent we got to replace our heavier car-camping one. It was only $99 on sale, but it weighs less than 5 pounds, was easy to put up and kept us dry, which is pretty much all I could ask for.

I saved money on the tent, but more than made up for it with the stove. I fell for the Jetboil a few weeks ago when a friend showed me his on Santa Cruz, and it was so worth it’s $130 price tag. Very light and compact, it heats water up incredibly fast and efficiently, has electric ignition, and is almost immune to wind.

With everything set up , we sat down to a meal and a serious game of Scrabble, as you can see in the lead photo. After some nice instant miso soup and an entree from Mountain House, we closed off the night with some martinis, served in genuine camping martini glasses:

During the night we were serenaded by coyotes, some very close, and there were smaller varmints snuffling in the nearby bushes. There was nobody else camping in La Jolla, so it was a bit spooky to be the only people for miles, especially with a full-moon a few nights before Halloween, but we both slept well.

It was a wonderful trip, topped off with a 5 mile hike to Mugu Peak the next morning. If you’re looking for a beautiful camping spot only 45 minutes from LA, I highly recommend La Jolla Valley. Call Point Mugu State Park on
805-488-5223 for more information, and for other Santa Monica Mountains campgrounds, check out these posts:
La Jolla Valley
Sycamore Canyon
Circle X
Santa Cruz Island

Funhouse Photo User Count: 1.954 total, 77 active. Still slowly but steadily growing.

Event Connector User Count: 88 total, 1 active. I haven’t been focusing on selling this idea to any organizers for the last week, and it needs some time and effort to persuade promoters to consider it.

PNG loading made simple

I need to load some images from disk, and use them as OpenGL textures. This is something I’ve reimplemented many times on different systems, and it’s always seemed more complicated than it should be.

The first choice is whether to use the native OS’s image loading libraries, or a cross-platform framework. Native libraries are a lot easier to start using, but make the code harder to port to other systems. Using an open-source framework makes your code much more portable, but it’s often a pain to find or build against the right version of the code, and you also have to deploy it to the end-users’ systems. I lost several hours wrestling with libpng a few weeks ago, since it refused to build on OS X, and I ended up yak-shaving my way down a dependency chain of other libraries it needed.

This seems overkill for most of the tasks I’m trying to accomplish, since I’m usually just trying to load a small, fixed set of images, which I’ve created and so can control their properties. This only uses a tiny part of most image loading libraries, since they’re designed to cope with reading and writing dozens of different formats, and hundreds of combinations of bit-depth, compression and other flags within each of those, along with lots of platform-specific optimizations to improve performance.

Happily, I recently came across the LodePNG project, which offers a simple, dependency-free way of loading PNGs. The PicoPNG module is a standalone C++ file that will load 8 bit, 4 channel PNGs, and is only 500 lines of code. This is perfect for my needs, there’s no extra installation hassles or dependencies, and it’s cross-platform. I also appreciate the elegant way that Lode Vandevenne packed his solution into such a small amount of code. It’s compact, but not obsfucated, very classy work.

Funhouse Photo User Count: 1,933 total, 78 active. Still getting quite a few users from the profile box add link, according to my statistics.

Event Connector User Count: 87 total, 4 active. Very quiet, I haven’t been actively chasing up any leads for a few days.

Jerry Mitcham, trailwork hero


Last night was the annual Santa Monica Mountains Trails Council board meeting and dinner. Jerry received the Hank Grateful award for the most time worked this year, with an astonishing 366 hours spent fixing up the trails! For just one project, he hiked miles out into the backcountry with a 30 pound post hammer to install over 70 signs. He’s not only a hard worker, he’s also a leader, both day-to-day as a crew boss, and working with the park agencies to make work trips to Santa Cruz Island happen. A good friend and lovely guy, I always look forward to working with him out in the mountains.

He’s not alone out there, many of the regulars clocked up over 200 hours each. Looking around the room that night, I was thinking about the difference all these volunteers make, even though most trail users will never know about them. They’re not doing it for recognition, money or out of obligation, they just take joy in getting things done, in improving the mountains they love.

The awards had the motto "The reward for work well done is the opportunity to do more". The happiest people I know are those who love what they do. I love being in a team full of them, whether it’s in the mountains or a conference room.


Implicit web philately


With Defrag fast approaching, I’ve been spending some cycles thinking about what the Implicit Web actually is, and where it’s going. When I’m staring at this sort of problem, a technique I find really useful is "stamp collecting". Gather as many examples as possible, list their important properties, group them into clusters and look at what patterns emerge.

Here’s my current list of Implicit Web services currently out there, with a couple that are on the borderline of the term. I’ve not got enough to meaningfully group them, so they’re alphabetical:

Adaptive Blue – More of a semantic web application, but they do offer a Firefox extension. Being client-based is a distinguishing feature of implicit web apps, since that’s the only way I know to get access to the user data needed.

Amazon – Their recommendation system is the grand-daddy of a lot of the apps that take raw information on user’s behavior, run some magic algorithms, and return something useful back to the customer. It’s a hard trick for most startups to repeat, since almost nobody has access to the Amazon’s breadth of data. This is why client-based solutions that can track behavior across many sites seem like the only practical solution. – A true implicit web app, they have client-based tracking of the user’s behavior, they piggy-back on other people’s applications to gather their data and use that to return Amazon-style recommendations. It does make me wonder about the ‘web’ part of the term though, since that seems to imply web browsing. Maybe ‘implicit internet’ would be more appropriate?

me.dium – Another app that fully fits the term. A unique feature is that they use the social graph to combine information from multiple users, which I think is a very promising area for implicit web applications. Being able to pool data from your friends is a great way of discovering relevant new content. – This is one I came across relatively recently. It’s a client-side app that monitors your browsing, and tailors a sports portal site to match your interests based on that data. What’s really interesting is that it’s aimed at a mainstream audience of sports fans, rather than geeky early adopters. I know from my game career that the sports audience is massive, and willing to pay for something ties into their passion, so I’ll be following its progress closely. The only audience I know that’s similar is music, and it’s relevant that the most successful implicit app so far,, tapped into that demand.

tape failure – This is a service I’ve only read about, but unfortunately their site seems to be down at the moment. They’re not an implicit web app at all, but it does seem like they have a good solution to the browsing data collection problem.

Let me know if you think I’m missing any. I may put together a page tracking new services, since I think we’re going to be getting a lot more over the next year.

Funhouse Photo User Count
: 1,916 total, 92 active. The proportion of profile-box adds was a bit higher this time, which is promising because it scales a bit more virally than the product directory.

Event Connector User Count: 84 total, 9 active. Not much happening on this front.

Accessing mail with Extended MAPI

In the first part of this series, I gave an overview of the APIs you can use to access Outlook mail, along with directions for how to get started with Visual Basic for Applications.

As I mentioned, using the Outlook Object Model/VBA is both slow and throws up a lot of security warnings. I need to process a lot of mail messages, and I need a language that will handle both heavy data processing and advanced graphics for the work I’ll need to do to turn the messages into a graph. This means I need a non-interpreted language with a lot of libraries, such as Java, C# or C++. Luckily, Extended MAPI is an old-school COM API that fits well with C++, and avoids the security restrictions.

One thing I didn’t realize about MAPI is that it’s usable even if you’re not running as an Outlook plugin. It’s designed to let third-party programs interact with the user’s mail store, and is available as long as Outlook has been installed on the system. This page is the most in-depth description and bibliography I’ve found.

The down-side of MAPI is that it’s COM, offers a lot of functionality, and has been implemented slightly differently by each version of Office. This means that using it can be tough, just getting to the ‘hello world’ level requires a lot of work, and past that there’s a lot of pits with spikes at the bottom to wander into. What I needed was a good code example to get me started.

Lucian Wischik came to the rescue, with some wonderful sample code demonstrating exactly how to use MAPI, along with some utility functions to simplify access, ample commentary explaining what was going on, and a completely public domain license for use. Amazingly, this is the third time in recent months that I’ve found what I was looking for on Lucian’s site. He also has the best samples on the web for writing a BHO without ATL, and hosting IE web rendering inside your own window. He must be accruing some serious good karma.

With Lucian’s help, it’s not too hard to create your own standalone executable that enumerates Outlook folders and items, and extracts information from them. Unfortunately, it’s not always possible to pull out the full HTML body of the message, but I’m mostly interested in the sender and receivers of each email which is easily accessible. It runs a lot faster than the equivalent VBA code, and doesn’t bring up any security dialogs.

Funhouse Photo User Count: 1,892 total, 101 active. Almost exactly the same as last night, so I must be looking at the same sample window in the stats.

Event Connector User Count: 82 total, 4 active. Also very similar.

So L.A.


I just talked to my neighbor, who’s having his back wall rebuilt after a car flew off the freeway into his pool. Liz spent an hour cleaning the ash out of our hot-tub. Dinner was an In-N-Out animal-style burger, but I’ll be working that off with a ride along the beach path in Santa Monica, after a 90 minute commute on the 405. This weekend, fires permitting, we’ll be camping with the coyotes, in the mountains north of Malibu.

I don’t stop to think about it as often as I should, but when I do it always blows my mind. As a kid in Britain I grew up watching this world on TV and dreaming. Now I’m living in it! There’s so many things and people I love and miss in the UK, but this feels like home.

Funhouse Photo User Count: 1,892 total, 109 active. It’s been good to see the active count increasing along with the total. I need to graph this again.

Event Connector User Count: 82 total, 5 active. Still picking up some random users from the directory, but no takers from event organizers.

Outlook plugin basics


My current mission to create a social graph from my mailbox. I need to get access to the raw data on my email habits, and the easiest platform for that is Outlook, since it offers a plugin API. I had no clue how to get started with it though, and most articles I could find assumed some level of familiarity with the technology, so here’s a guide covering what I discovered on the absolute basics.

The first thing I discovered is, like standards, that the great thing about Outlook APIs is that there’s so many to choose from! The second discovery was that the flood of email viruses using those APIs a few years ago led Microsoft to barricade most of them behind heavy security. Luckily, there’s a lot of people writing Outlook code, and so there’s a lot of examples and resources to learn from. The best starting point is, but plain google searching will usually throw up some answers to even fairly obscure questions.

Collaboration Data Objects (CDO)
are a legacy interface API, and using them will often bring up scary security dialogs, since Outlook assumes you’re doing something malicious.

The Outlook Object Model (OOM) is the most common way of working with the data held by Outlook. It the interface you use by default when writing Visual Basic for Applications macros within the app, it’s well documented and supported, though you have to be careful for differences between app versions. Unfortunately for us, it’s also hedged in by security warning dialogs and restrictions. Because you generally have to make an API call for each mail item you want to deal with, it can be very slow if you want to handle thousands of them at once.

Extended MAPI is an advanced interface to the mail store in Outlook, and it’s mostly commonly used from from C++ or other compiled languages. The good news is that you’ve got access to almost any information, with no security UI. The bad news is that it’s not well documented, and is tricky to work with. Because you can access lots of items at once, without the overhead of an interpreted call every time, it can be a lot faster when you’re working with large numbers of emails.

Redemption is a third-party library that combines a lot of the advantages of Extended MAPI and OOM. Under the hood it uses MAPI, and so it’s fast and avoids security warnings, but it exposes an interface that’s a carbon copy of OOM, which makes it simple to use. It’s a real feat of engineering by Dmitry Streblechenko, and he’s also been very prolific in answering Outlook programming questions. At only $200 for an unlimited distribution version, it’s a great deal.

Getting started prototyping your code is as simple as pressing Alt+F11 in Outlook. This will bring up a separate VBA code window. If you select the session in the left pane, then take any of the simple examples out there and paste them in, you should be able to run them by hitting F5.

To make Redemption available, run its installer and then it should appear in the code editor’s Tools->References dialog box as an option. Then just reference its objects like any of the built-in types.

I’ve managed to create a simple prototype that loops through all the messages in a folder and outputs the basic information I need to build my graph to a text file. It’s slow, probably only a few dozen messages a second, but it should be good enough to let me create the foundation for a social graph. I’m moving over several years worth of email now, so I’ll have a good data-set to work with.

Funhouse Photo User Count
: 1,847 total, 93 active. I had a new review, requesting larger photos. That’s an interesting idea, if more people had photos in their albums that could be a compelling feature. I also saw the stats showing more people removing than adding the app, which doesn’t seem to match the total figures!

Event Connector User Count
: 77 total, 4 active. Still very quiet.