Photo by Scott Ableman

I've just finished The Path Between the Seas, the story of the building of the Panama Canal by David McCullough. I was left in awe of the builder's achievements, and the price they paid in lives. It really was a heroic age of engineering and almost every page had something that made me think about my own work. Here's some of my favorites.

"Now, boys, we have got her done, let's start her up and see why she doesn't work"

Release early, release often for the 1900's. Sometimes the only way to find out is to try it and see, and this was a big difference between the French and US attempts at building the canal. The Americans were brought up in a culture that valued improvisation and experimentation at all levels, the French found the flexibility needed to cope with brand-new challenges a lot harder to come by.

"Engineers are sometimes the least practical of men, they may be attracted by difficulties"

Before malaria and yellow-fever were brought under control the death-rate amongst the canal builders was horrendous. Despite that, the challenge was so compelling that the American engineers tasked with evaluating the project's potential were irresistibly drawn to it. I recognize this in my own work- if there's a powerful but hard-to-master solution to a issue (compiling Cassandra anyone?), I'll find myself inventing reasons to use it rather than a more mundane alternative, just because of how interesting the problems would be.

"The diagrams were as simple as illustrations in a childs primer, conveying their message at a glance and easy to remember. They were an inspiration, Hanna saw instantly. The inevitable problem with technical reports, with any arguments based on technical data, was that few would read them"

Communication is the key. Most people have far less time and patience than you imagine, so if you're going to persuade them you need to expend a tremendous amount of time and energy simplifying your story. The managers of the canal project spent years ignoring the overwhelming evidence and scientific consensus that mosquitoes spread malaria and yellow fever, a blindness that cost thousands of lives. Never underestimate how hard someone's established opinions will be to budge.

"You won't get fired if you do something, you will if you don't do anything. Do something if it is wrong, for you can correct that, but there is no way to correct nothing"

I don't know that my bias towards action always leads me to the wisest path (I didn't expect what would happen when I poked Facebook with a stick), but I get incredibly frustrated when I can't make progress. I'd rather screw up, take my knocks and learn a lesson than be stuck in indecision. I love these instructions to a subordinate, it perfectly captures the attitude I look for in the people I work with.

"Suppose after all that should prove to be right, and it all ends in your butterflies and morlocks. That doesn't matter now. The effort's real. It's worth going on with. It's worth it-even then" – Teddy Roosevelt in a meeting with HG Wells

I consider myself a cheerful pessimist. I've been through enough that I know how steep the odds of success are, but I've made a choice that even a hopeless fight in a good cause is worthwhile.

"Bishop began publishing weekly excavation statistics for individual steam shovels and dredges, and at once a fierce rivalry resulted, the gain in output becoming apparent almost immediately. 'It wasn't so hard before they began printing the Canal Record' a steam-shovel man explained to a writer for the Saturday Evening Post. "We were going along, doing what we thought was a fair day's work … [but then] away we went like a pack of idiots trying to get records for ourselves"

Statistics are incredibly powerful motivators, we're driven to measure ourselves against others so just exposing a new metric can radically change people's behavior. The tricky thing with our engineering world is that output is much harder to quantify. If you look at lines-of-code per day you'll be rewarding hacky programmers who cut and paste, not the elegant coders able to reduce complexity.

"Even New Englanders grow almost human among their broad-minded fellow-countrymen. Any northerner can say 'nigger' as glibly as a Carolinian, and growl if one of them steps on his shadow"

The white workface was tiny compared to the number of black laborers. They were given an astonishing range of perks from free housing to social clubs while the blacks were left to clear a home in the jungle.

It's worth looking around every now and again and to really see the people who are dealing with the less glamorous side of our work, especially now we're outsourcing so much overseas. Are there potential stars we're overlooking because they're not the people we expect to excel? Are we sharing enough of our success with the people working hard for our company, even when their skills aren't as prestigious?

Photo by Sam Cornwell

Eddy's sofa and the nightmare of a single global places register – The large web companies want to own a monopoly on some important data source, it's their strategy to lock in long-term defensible revenue. This scenario lays out how seductive a bait-and-switch approach could be for geo data, persuading everyone to input their information and then later putting up pay-walls.

Why do women leave science and engineering – A new study argues that it's not the traditional culprits, but poor promotion prospects that drive women from our industry

Hacker News on Why nobody understands your visualization – As always HN has a high standard of discussion, and pulls out some great war stories and further reading

My contrarian stance on Facebook privacy – I found myself nodding constantly as I read Tim O'Reilly's take on Facebook's recent troubles. Despite my own legal tussles with the behemoth, and my dismay at their cavalier attitude to user privacy, I think there still needs to be space for innovation

Am I human? – Adding CAPTCHA to graffiti. I wish they'd taken this further and caught some people noticing and reacting to the additions

I love visualizations, but they keep letting me down. I think I've built something that tells a crystal clear story, only to see people's eyes glaze over when they sit down in front of it. Why don't most of my pictures communicate the story I want to tell? And how was my map of Facebook connections different, what did I learn from its success?

The obvious place to look is Edward Tufte's work, but even as a fan I still find myself producing unintelligible visual explanations. Instead, I've found it more useful to think about the problem in a much simpler way. I use a visual vocabulary that nobody else understands. Without really noticing, I've invented a private language that makes perfect sense to me but conveys nothing to anyone else. So how do I make myself understood?

Keep it simple

There really is a visual vocabulary that we all learn. Our visual vocabulary is not as obvious as the verbal one because acquiring it is a much more informal process. There's no dictionaries or classes dedicated to understanding diagrams, so I had the unconscious idea that they just automatically make sense. This is seductive because our culture ensures that most of us do readily comprehend common charts like maps and bar graphs, which in fact would be incomprehensible to most of our ancestors. As the Tour through the Visualization Zoo points out "Although a map may seem a natural way to visualize geographical data, it has a long and rich history of design". Every familiar form (even the map) had to be invented, and only became widely-understood after it had proved its usefulness over a long period of time.

I try to remember that and stay humble whenever I'm tempted by a shiny new way of presenting data. There's probably about six basic ways of presenting data that everyone understands, colored maps (choropleth, hate the name!), points on maps, bar graphs, line graphs, pie charts and scatter graphs. If the new technique doesn't reuse people's understanding of those basic elements, 99% of people will look blank and move on. A few highly motivated people might puzzle out what you're showing, but if you want to tell a story to a wide audience, keep it painfully simple.

Label and annotate

I'd released an earlier version of my Facebook connections map two weeks before the version that became popular. It shows exactly the same connections, but without any coloring, labels or descriptions. The crucial breakthrough in reaching a mass audience was adding that extra guidance, making it very clear what I thought the diagram showed. I could see the same story in the spider web of connections, but that was because it was in a visual language I understood. By highlighting and naming the different areas I was able to tell that story to other people who would otherwise be baffled by the mass of lines.

Animate and interact

There's a massive untapped element that everyone understands: time. Before computers came along, animated graphs were almost unheard of, so they are absent from the traditional literature on creating good diagrams. The tools are still lagging behind the possibilities, but there's a rich world of vocabulary and metaphors to draw on. Our brains come packed with hardware to translate moving images into meaning and Saturday morning cartoons help train that capacity! If you can take common elements like bar graphs and animate them, you've got a whole new axis to display information on.

Even better, you can pack more information in without confusing people by making your charts interactive. If someone selects an element, you can focus on that dimension of your data in much more detail, something that's impossible when you're communicating through a sheet of pulped wood.

Show me

You may have noticed that my last point doesn't fit, I didn't use time or animation in my Facebook visualization. That was because I didn't have the technology I needed, so I have no proof to demonstrate that I'm right. I'm working to remedy that problem, so stay tuned for my next project.

Photo by J-ster

Try Gnip – My friends at Gnip have just launched a free trial service. Making it all self-serve radically lowers the barriers to just giving it a try, so if you're at all interested in sucking down masses of data from Twitter, Flickr, etc, you really should give it a shot.

The Hotlist – There's tonnes of information about what our friends are up to flowing around us these days, but it's far too time-consuming to actually make sense of it all. That's why I love this mashup of your social graph, Facebook events and a map to help you discover events you should care about.

Jumppost – It's really hard to uncover the information you need to find good apartment rentals, especially if you're looking more than a few weeks ahead of your moving date. These folks have figured out that existing tenants know when their lease is up, and are willing to pay them up to $500 to pass that information along to apartment hunters.

Everlater books – My old Techstars colleagues Nate, Natty and Ryan have been doing an awesome job with their service that makes it much simpler to share your travel memories than the alternatives of email, blogs or Flickr albums. Now they've made it easy to create books from your trip diaries, and the quality of the examples I got to handle was impressive. If you order one today, you'll get free shipping too.

Super-simple Storage Service – I've been frustrated by how much complexity is introduced into my processing pipelines by actually having to read data, so S4's new write-only storage engine promises to radically simplify my code. At $12 a year for a terabyte of data the price is right, and AT&T's adoption of it for their customer complaint database proves it's ready for enterprise use.

Photo by Johnny Grim

"How am I in danger? Do people really care about what I post – like and
dislike on a social networking site? If so, what are they going to do
with the information? I don't get it."

This question came up in the comments of my blog, and though it's very simple, the answer's surprisingly complex and brings up much deeper philosophical questions.

The short answer is that you're in no danger right now, despite all the gnashing of teeth and wailing in the tech community. There's no evidence that anyone's using this information for malicious purposes, just as I've seen no actual burglars using the information in Please Rob Me.

So why are the geeks so upset? They're looking down the road and imagining all the bad things that the people wearing Black Hats will be able to do once they figure out what a bonanza of information is being released. Do you remember in the 90's when techies were hating on Windows for its poor security model? That seemed pretty esoteric for ordinary people because it didn't cause many problems in their day-to-day usage. The next decade was when those bad decisions about the security architecture became important, as viruses and malware became far more common, and the measures to prevent them became a lot more burdensome. The geeks were proved right, you can't start with a shoddy security model and just patch it into something secure.

I think the inelegance of Facebook's approach is what makes engineers' skin crawl. The model they use to prevent your information leaking out is a mess, both from the API side and in the UI. This makes it almost certain that there's unintended holes that leak information that even Facebook aren't aware they're revealing, and ensures users have no clue about what they're opening up to the world.

Fueling the anger is the feeling that Facebook is being deceptive in how they change their privacy model. They appear to believe there's a simple trade-off between making money and keeping users happy, and have apparently decided that they're in a strong enough position to ignore user complaints in order to increase their revenue. They're making information public because they want Google Juice. The more user-generated content they have on the public web, the more visitors from search engines they'll get, and the more important it will be for companies to have Facebook pages and advertising.

In practical terms, why is the information they're revealing important? Here's some of the scenarios that dance through geek's heads:

Embarrassment: There's a lot of personal information we'd rather keep to ourselves that might be revealed by our fan choices or friendships. You fan a gay club, and a homophobic potential employer spots that. Your ex-partner's divorce lawyer spots you're a fan of 'partying', and uses that as evidence against you in a child custody battle. Someone with a grudge targets your friends and family for harassment.

Big Brother: Social tools played an important part in the Green uprising in Iran, but you can bet your bottom dollar that there's now people within the regime using the same tools to track down dissidents. There's a lot of people within Iran who are fans of Mousavi, and since people generally use their real names on Facebook they could easily be found. I actually removed detailed data from FanPageAnalytics for Iran, Burma and North Korea because I was worried about this sort of usage.

Criminals: I'm skeptical that social network information will help traditional criminals, but there's a massive world of phishers, scammers and identity thieves I can see learning to use what's being revealed. If you got an email that appeared to be from one of your friends, said hello by name and giving you a link to something you were interested in, wouldn't you be a lot more likely to click on it? Facebook's starting to reveal the information criminals need to personalize social engineering attacks like phishing emails, it's just that the bad guys don't have the sophistication to use it yet.

So, don't panic, but pay attention to what Facebook's doing. In the short term the biggest security issue on the site is still the spread of traditional Windows viruses and malware so keeping your virus checkers up to date should be your first priority. Long term, we need to figure out what information we want to reveal, rather than letting Facebook decide for us.

Photo from Marquette University

I'm fascinated by statistics, and a lot of my work has revolved around trying to analyze and visualize online activity, whether it's Twitter conversations, email behavior or Facebook friendships. These have generated a lot of interest, but it's been hard to see how to convert that into enough revenue to create a real business.

The fundamental problem is that in most cases the statistics aren't solving a painful problem for anyone. We all love to look at ourselves in the mirror, and services that analyze our online behavior satisfy that craving, but there's seldom enough to justify a subscription or purchase. There's also the problem of driving repeat visits. Many meaningful statistics are very static, so the second visit to a site will often show exactly the same information as the first, discouraging continued engagement.

As a practical example, look at Xobni. I love what they're doing, and they initially launched with a lot of in-depth statistics about your email behavior, but in subsequent releases de-emphasized those in favor of productivity enhancements to Outlook. That's been successful for them and they're expanding back into analytics again, but it shows what a problematic driver statistics can be.

There is a tried-and-tested path to building a business around analytics though. I think of it as the Feedburner model, since they're the first company I encountered using it. They offered analytics as the carrot to give users a reason to sign up to the service, and then monetized by inserting ads into the RSS feeds they now controlled. I'm addicted to my Feedburner stats, but I'd never pay for them, so this was a great way of getting revenue out of a free service.

More generally it doesn't have to be ad-based, you just need a business model that benefits from access to a large audience of engaged users. Can you up-sell premium services that appeal to the same market? Can you get permission to pass their contact information to other companies they might be interested in, and get paid for sales leads? Is building an appealing analytics package just a good marketing investment, driving traffic to your site more cheaply than conventional advertising?

If you are interested in that approach, here's a couple of tips. First, try to show users something as soon as possible. In an ideal world they arrive at your page and immediately see a graph that tells them something interesting about themselves or something they relate too. Typically this isn't achievable, but at the very least have a single step where they enter an email address, twitter name, etc and then within a few seconds get some information. You should also show an example of what they will get on the landing page. These techniques reduced my bounce rate massively, never overestimate people's patience, you constantly need to be convincing them to spend time navigating your site.

The second key is presenting your statistics in an actionable way. If you can not only tell a user something interesting, but cause them to do something based on that information, then your chances of a repeat visit shoot way up. Feedburner has an 'Optimize' tab that guides you through ways of increasing your traffic. I found that changing from just showing your most-frequently-contacted friends to sending a report of the people you used to talk to and haven't for a while ('Losing touch report') and giving them a link to email each person alongside the list turned it from an 'oh, that's nice' to a must-have.

If you're as addicted to statistics as I am, but despairing about turning it into a business, it's worth thinking laterally. Analytics are a great carrot, can you use them to super-charge another business model?

Photo by Ucumari

The Institute for Unnecessary Research – Art, science and magic, makes me wish I was at NetSci to hear about Cybernetic Bacteria

Pentaho – Some friends have been producing amazing results with this open-source business intelligence package, it's a really impressive framework for exploring your data

An analog video synthesizer – I'm fascinated by the outer reaches of video technology, and I love this application of old-school analog technology to video stream by LZX

CreditUnions.me – I've ranted about this before, but if you're in the US and want your money to be lent out to local businesses rather than pumped into financial speculation and prefer customer service from actual humans, you should check out this site to find a nearby credit union

Karmasphere – I haven't spent much time with it, but so far this friendly interface to Hadoop looks very promising. Elastic MapReduce does a good job of making Hadoop available to the masses, but this goes a long way to make it a lot less intimidating too

One of the joys of hiking in Britain is that you can end every day with a pint of beer. Even the remotest hamlets will have a pub, even if there's barely enough inhabitants to staff it, let alone provide customers. To take advantage of this, we set out on the mother of all pub crawls – 40 miles from bar to bar along the West Highland Way.

Sitting in Lauders pub in the center of Glasgow, I savored my first pint of '80 shilling', the national beer that I've never seen outside of Scotland. It's closest to what you'd call a 'bitter' in England, dark, rich and not fizzy like the typical 'lager' beers in the US. Every brewery has its own version and you'll rarely go wrong in Scotland if you ask for 'a pint of 80'. As we enjoyed our pints, we were entertained by a group of 12 year-olds smoking cigarettes and spattering each other by stamping on ketchup packets.

The next day, we caught the train to the start of our hike, Crianlarich, about half-way along the Way from its beginnings in Glasgow. The station itself had the world's tiniest tea-room squeezed into what appeared to be a converted toilet cubicle on the platform, but fortified by visions of a luke-warm pint at the end of the day we started out on our hike.

Though our first stage was only six miles, we stumbled over the sort of history that tends to accumulate in a small country that's been inhabited for 8,000 years. We passed a pond that Robert the Bruce had thrown away his sword whilst being pursued, and slightly later where he stopped and defeated the pursuers. From my memories of Dundee's nightlife, I'm guessing he just head-butted them into submission.

We rapidly realized that most people weren't carrying 30 pound packs, and instead had taken advantage of the van shuttle services to take their gear between each stop. That just seemed like cheating, but then we weren't the real hard-core, carrying all their camping gear in even heavier packs, or even running the whole 95 miles in under 24 hours!

Just before we arrived in Tyndrum, a large patch of bare earth accompanied a cheery sign explaining that the are had been used for hundreds of years to crush lead ore using child labor, leaving it poisoned and lifeless, and pointing just off the trail to our B&B. Happily it was upstream from the old leadworks, but it was yet another reason to stick to refreshments other than tap-water.

Wandering bedraggled into Glengarry House, we were immediately greeted by the owners Ellen and Pat. Wonderfully helpful, they magicked up a welcome cup of tea as we got settled in our room, and we started to regret our plans to eat out that night as we smelt the meal they prepared for their other guests. Steeling ourselves, we wandered half a mile down the way to Tyndrum proper, and made ourselves at home in Paddy's Bar. My McEwans 80 shilling hit the spot, but the place itself was fairly anonymous, a bit like a converted portakabin. Moving across the road for food, The Real Food Cafe was stupendous, with ana amazing menu of lovingly prepared versions of British classics like fish and chips, all using local, organic and even gluten-free ingredients. The seating was at two long bars, which led to a lot of socialising between the parties.

Next morning we chatted to our hosts a little more (apparently April is a much quieter month than May, with just as good weather) we strapped on our packs and, after a brief diversion to The Green Welly Stop for postcards, we headed for our next stop at Bridge of Orchy. Again, this was a short stage, only 7 miles, but it still left our feet aching as we came down the hill into the village. I'd booked us into the main rooms at the Bridge of Orchy Hotel (they also have a bunkhouse), and the staff treated us like kings. Almost entirely staffed by South Africans, both my pint of Belhaven Best and the following meal of salmon were superb and delivered with great ceremony. The barmaid offered me a tasting glass of some of the other beers while we were browsing, and Liz was handed an impressive data sheet on the different characteristics of the whiskies after she asked the bar tender's advice. The receptionist enthusiastically told us "That's sensational!" after she heard that we'd done an extra reconnaissance hike up the hill that evening, which became our phrase of the day.

Day 3 was the first of the tough hikes, 12 miles to the Kingshouse pub with 1500 feet of elevation. We had a slog up and over the ridge we'd explored the night before, passing the Inveroran hotel and campground, then a seemingly never-ending march up a sloping old drovers road, headed to Ba Bridge. We were apprehensive about this section because we'd seen signs warning that the track was going to be used as part of the Six Day Trials motorbike event, and we did get passed by a few, but they were remarkably quiet, more like mopeds than the Harleys we used to live next door to in California. It was fascinating to watch 'guys trying to ride a motorbike up a waterfall' as Liz put it, and I was actually glad we caught them up close.

The Kingshouse pub appears as you come over the final ridge, and it looks deceptively close. When we finally arrived, we both nursed a Calder's 80 Shilling and called for a taxi to our B&B, since Kingshouse is alone in the middle of a stunningly beautiful stretch of moorland and they'd been booked up even 5 months ago! The taxi was going to take a while to get to us, so we ended up having a second pint which led to a very chatty ride.

Kinlochleven itself is in a valley at the head of a loch, surrounded by steep hills wooded with beech trees that looked golden with the sun on their spring leaves. The view took my breath away, and our room at The Highland Getaway had a back window looking out onto a wide stream. Wandering around the town, we stopped at the Tailrace Inn for dinner and another pint. It felt like a traditional British pub, my Tennant's Ember 80 shilling was almost spicy, but Liz's request for whiskies was met by the barman with a scowl and a gesture at the shelf behind him. A final whisky at the Harlequin Cafe attached to our B&B was a lot more pleasant, with the dark wood and red wallpaper that every pub needs.

We skipped the Kingshouse to Kinlochleven stage of the way, missing out on the Devils Staircase (Beelzebub gets all the cool geography), but our final day was the real test. 16 miles and 2,000 feet of elevation to make it to Fort William, with our heavy packs on. It was just as painful as we'd imagined, but we finally stumbled into Fort William and the end of the way. With astonishing lack of foresight, nobody had built a pub there, and we ended up walking all the way to the town center until we found liquid refreshment. The Crofter gave me another chance to try McEwans, while Liz fell back to the old favorite of Guinness. Wandering down the High Street, I discovered that the little gift shops selling figurines had apparently progressed onto Stripper Fairies since my last visit:

I had high hopes for our final lodging, the Cruachan Hotel. As Liz said when she saw its website "it looks like a castle!", and I'd gone crazy and spent $200 for the night. I had my first misgivings when the sign out the front advertised vacancies and $75 rooms for two. That first impression was borne out by the pokey room that looked like a time capsule from 1973, complete with no toilet lid, a single pillow each that was flat as a pancake, mould in the bathroom and strange stains on the wall heater.

We really didn't care though, we'd made it! It was an amazing trip, I just wish we could have had more time and managed the whole 95 miles. The West Highland Way is a hidden treasure, I was surprised at how few people we encountered, and almost everyone was local. If you're considering a visit to Scotland and want to experience that amazing landscape up close, I can't think of a better vacation. Just don't forget to check out the '80'!

Photo by Jeff the Trojan

I'm flying home from the UK today (volcanoes permitting) so if you're waiting for a reply to your email I should be caught up soon.

Facebook files criminal charges against startup for violating terms of service – I think Arvind nails it with his analysis of Facebook's real problem. They're scared that they're losing the trust of their users, and they've fallen back to legal bullying rather than trying to solve the thorny issues that caused it.

Top 10 reasons you should quit Facebook – The incredible popularity of Dan Yoder's article makes it clear  there's deep concern about Facebook in the tech community. I'm still amazed by everything they've achieved, and don't want them to jump the shark, but if this starts resonate with people outside the bubble then they're in deep trouble.

Amazon S3 file deletion fail – Amazon make it very hard to delete large amounts of data once you've uploaded it. You can't even delete a bucket without removing all of the contents first, which makes no sense at all. Of course, you could always delete you account, and somehow they can remove the data then!

Location-based art: Audio Graffiti – Art is an incredibly underrated driver of technology, artists with a technical bent often come up with amazingly innovative uses of new tools, and this presentation made me miss my days of hacking on VJ software.

Jedi vs Tescos – I spent three years working for Tescos on the checkouts, and learned an amazing amount about customer service from one of the few UK stores to care about it. I thought the spokesman's comments at the bottom were pitch-perfect, "If Jedi walk around our stores with their hoods on, they'll miss lots of special offers". (via Overlawyered)

Photo by Wink

Tribalytic – My friends Alex and Tim have been doing some fascinating work applying statistical analysis to Twitter conversations. Their 'space shuttle control panel' interface can be a bit off-putting, but if you dig down you can see insights into the Chirp talks that prompted the most traffic and spot Binh from Klout looking forward to the after-party.

Breadcrumbs – Kate McKinley's created a great demonstration of how many different ways there are to store data about a user, so advertisers can identify them as they move around the web. She's also produced a good paper covering the details, which reminds me of Arvind's work on 'super-cookies'.

Spinn3r have announced in their email newsletter that they're now crawling public Facebook pages and making the results available as a feed for their commercial subscribers. "We're indexing all Facebook public pages, which do not require login, including public fan pages and their wall posts, videos, albums and pictures. We also index Facebook public groups including topics and the comments responding to these topics. The current volume is in excess of 50k permalinks and 30k comments per hour."

Sendgrid have received $5m in VC funding. I went through Techstars this summer with Isaac and Jose, and loved their quiet focus on solving a vital problem, helping companies reach their email subscribers without ending up in the spam box. They have been kicking ass and earning revenue, and this injection of cash will help them reach even higher.

"Plates of Spaghetti" graphs – Almost all visualizations are terrible at communicating information, but are often fantastic marketing devices, drawing people into looking into the source data. I like the quote "The “data visualizations of the year” really are impressive if you think
of them as super-cool illustrations (replacements for the usual photos
or drawings that might accompany a newspaper or magazine article) rather
than as visual displays of quantitative information". I've long been mulling a post entitled "Most visualizations are useless" (including mine!)