Photo by Ocean of Stars

Changing behavior – it ain't easy – The fundamental challenge for any startup is how to change people's behavior so that they begin using your service. This article is about energy conservation, but everything in it applies to us too. The bottom line is that "changing people's habits in a systematic way turns out to be a painstaking, labor-intensive undertaking". The solution sure sounds like a lean-startup approach:

"You research the barriers and benefits that are most salient or motivating. You don't guess at them — as people all too frequently do — because they're often surprising or counterintuitive. You do the research. Then you select promising methods to reduce barriers and increase benefits and run a small pilot program; you keep at that until you find what works. Then, and only then, you scale up"

Steal this heatmap – Thanks to Lauren Kirchner at the Columbia Journalism Review for this great writeup of how OpenHeatMap can help journalists

Probabilistic Data Structures and Breaking Down Big Sequence Data – A great talk on using tools like Bloom filters to handle problems at a massive scale, with slides available here

Eye on Earth – An intriguing experiment in showing pollution data on an interactive online map, funded by the EU. I really like the concept, but I was somewhat baffled by the interface.

A world of tweets – A simple but compelling view of Twitter messages from around the world. I particularly like the animation showing incoming tweets, that helps the viewer make sense of what's happening.

Photo by Sally Crossthwaite

A lot of people have got very frustrated with the tendency for successful startup founders to embellish their company's creation myth until it becomes an outright lie – Pez dispensers anyone? I'm driven crazy by the misinformation floating around too, but I'm wary of destroying the beneficial aspects of the startup culture in an attempt to stamp out the lies. As Tom Evslin puts it "nothing great has been accomplished without irrational exuberance". If founders were truly sensible, all of us would be squirming our way up the corporate ladder instead of trying to build castles in the sky, the probabilities of success are way better.

These thoughts came to the front of my mind when I read The Grandeur of Glory, an exploration of the role of stories based around a research paper on sheep foraging patterns. The conclusion of the study was that you need a certain number of 'risk-taking' sheep who will wander widely to find new patches of grass, along with a large population of more sedentary animals to safely graze the known resources. What's intriguing about stories in human cultures is that they're almost exclusively dedicated to lionizing risk-takers. This might seem a given, but there's no fundamental reason why we should find Achilles a more interesting person than someone who's spent an uneventful forty years in accounting. My favorite definition of drama is simply "There's a problem". Our stories all center around characters tackling those problems, people with something at stake, taking risks, and Brendan advances a theory about why:

"Does this suggest that humans are too predisposed toward meekness, so that we require cultural encouragement to develop a sufficient number of risk takers to sustain the species?"

This feels intuitively right for the startup world. Everybody in positions of power has an interest in encouraging young entrepreneurs to jump in and take risks, so we all tell stories that at the very least involve some creative editing to paint a rosy picture of the process. The obvious downside is that we may be driving people with slim hopes of success into sinking their life savings and endangering their relationships in pursuit of a mirage.

To my mind, that's a price we have to pay. I grew up in Britain with a much more 'sensible' culture, and the priority given to safety drove me crazy. I need some risk in my life, and startups are a great way of channeling that tendency in a way that hopefully benefits society at large. The romanticization and mythology that surrounds the work that I do is definitely part of why I stick at it. I love having interesting stories to tell, it means I get something from even the most abject failure.

My hope is that we can unearth more complex, true-to-life tales from the startup world. I find it a lot more inspiring to hear from someone who's been repeatedly knocked down, and picked themselves up every time, than a sanitized tale from a founder who seemed to stumble across a winning formula with no false starts. Encouraging more founders to blog would be a good start, Tim Bull's site is a great example of how this can work. What we shouldn't aim for is the eradication of the whole mythology that has emerged around startups. It's actually the main reason we have an entrepreneurial community here in the US. The stories we tell each other (and ourselves) may be misleading, but they're necessary.

Photo by Tipiro

Carlos Ble's recent post about his experiences with Google AppEngine has made quite an impact. I found myself nodding along to almost all his points, I've evaluated GAE and found it unsuitable for my data processing work, largely because of the time limits it imposes. This isn't really a criticism of the system, it's just not the right tool for the job. Most of the limitations Carlos hit can be found by studying the documentation and mocking up a few test apps.

The problem is, the cloud has been oversold. I'm an enthusiast, I couldn't have done most of what I've done in the past two years without EC2, but it's no magic bullet. The cloud is like any other software engineering framework, the price you pay for the added convenience is an extra dependency in your stack. Here's the drawbacks:

Reliability. The hardest problems to track down are failures that are caused by the underlying virtualization infrastructure. Carlos discusses the outages he hit, but it's not just Google, this is a general issue with all cloud-based services. Early in OpenHeatMap's development one of Amazon's load balancers failed mysteriously, killing my site.

Apathy. Neither Google nor Amazon would really care if their cloud services divisions disappeared. Their revenue is dwarfed by the rest of their business, they're primarily a strategic bet and a way of re-using their existing expertise. In practice this means that they sit low on the corporate totem pole and don't have the resources or will to respond actively to issues. My load balancer problem never got resolved, despite continuously harassing the forums, sending emails to support, even tracking down AWS engineers at conferences. They just blew me off, even though I've spent $1000+ with them every month for the last two years. It's not just me, check out Jud's experiences trying to delete millions of files from S3.

Why not just buy premium support? There are cases where I'd consider it, but in my experience companies that are useless at free support don't magically become awesome once you hand them more money. It's baked into the culture – instead of ignoring your emails, they'll fire back detailed replies giving you 16 point checklists that will take you two days to complete, and still don't resolve the problem. Fundamentally, you don't have a 'throat you can choke' to get real help.

Conformity. If you use AppEngine for simple page serving and data entry forms you'll be fine. Amazon is great for running your own LAMP server. That's what these services were designed for and lots of people are using them that way, so you're unlikely to hit many problems. It's once you stray off that well-lit path that you stumble upon issues that the system designers hadn't anticipated, and bugs that escaped their testing.

I'm still using EC2, but trying to keep my usage as simple as possible. I've spoken to several startups recently who are trying to figure out how to host their services and laid out my experiences. At least one of them has gone for Rackspace, primarily for the support. I've considered that, unlike the big players it's the central focus of their business and so they're unlikely to be apathetic. The downside is that there's a lot fewer people using them, making it harder to find solutions and tutorials.

I love the cloud, it's made things possible I could barely imagine a few years ago, but there's no free lunch. Before you build your business around a cloud infrastructure, make sure you've factored the tradeoffs into your plan.

I spent most of my career working in the desktop world, and despite the obvious shift towards web-based applications, there are still hundreds of millions of users who rely on Outlook. I was one of those users for a decade, and I still haven't found anything online that functions nearly so well as an integrated communications hub. It's a shame that desktop-only startups don't get their fair share of attention, so I was pleased to see ClearContext's latest Outlook plugin get some love from GigaOm.

I've known their founder Deva Hazarika for several years, and was initially intrigued when I learned he'd raised his seed funding by playing poker! It's an unusual company in a lot of other ways too. They have an old-fashioned business model in that they are focused on making money through selling downloadable software to customers. What a crazy idea, eh? It's led them to be fanatical about improving their product, since they have to wow their end users in order to make any money.

The Personal edition they've just launched is interesting because it's their first free offering, and works as a showcase for some of the cool features they add on to Outlook. In particular they bring a priority inbox-like experience to the system, along with some very handy pre-populated rules for recognizing and filing common bacn emails like Facebook notifications.

Anyway, if you're an Exchange user looking jealously at the latest Gmail Labs creations, give ClearContext a try, they add a lot to Outlook.

Since my apartment hunting was over more quickly than I expected, and I’ve been enjoying the San Francisco food far too much, so I decided to go for a bike ride this morning. With the Golden Gate bridge looming, how could I resist? The guys at Blazing Saddles gave me some general directions when I rented my bike, but there were a few sections that were tricky to follow, so I’ve mapped the route I ended up taking from downtown San Francisco, and give my own directions below. I rode from the city to Sausalito and back again, with a nice lunch spot at the turn-around point, and it was about 22 miles total with some non-trivial hills.

For convenience I’ve started the map at Powell Street BART station, but I actually did a more complex route through downtown, since I had errands to run. There’s a great bike lane down Market Street though, so I recommend taking that south until you get to Page Street. This is then a straight shot to Golden Gate park, and it’s mostly quiet, residential and bike friendly, though you will need to navigate frequent stop signs. In the park itself you take bike paths through the north-east corner to Aguello Boulevard. That heads north towards the massive Presidio park, and it can be a little tough dealing with the traffic despite the bike lane. Once in the park, continue along Aguello until Washington splits off to the left. Washington will take you up and over the ridge, and then merges into Lincoln. A little way down Lincoln, turn left on Merchant, and follow that until you’re approaching the PCH/101.

This was the hardest part of the ride for me to follow. I knew that on weekends, the western sidewalk of the bridge was reserved for cyclists, but I had a hell of a time figuring out how to get there. Eventually I worked out you had to take the tunnel under the roadway from Merchant, then bike up on the east side past the gift shop, then follow another tunnelled path under the roadway back to the western side and the bridge’s sidewalk.

The bridge itself was fantastic to bike on, even on a busy Sunday morning. The weather was clear and sunny with breath-taking views across the water, the grade was steady and even the most aggressive bikers had plenty of room to swerve around tourists. At the other end you come off into a parking lot, and can then head onto Alexander Avenue to Sausalito. This can be a bit hairy, since the road is narrow and steep in sections, but the drivers seemed quite bike-aware thankfully. In Sausalito itself, you end up riding along the waterfront. There’s a lot of choices for refreshments, but I was intrigued by a tiny hole-in-the-wall place that simply promised ‘Hamburgers’:

Yelp gave it a thumbs-up, so I ventured in for a cheeseburger and fries. Man, that tasted good! They’re apparently a bit Soup-Nazi in their customer service, but I didn’t see anything other than slight surliness. It was lucky I got there early because they soon had a queue out the door, so I guess I’m not the only one to be hooked by their food. There’s almost no space to eat inside the restaurant, but the park across the road had a gorgeous view over the bay and plenty of benches.

Blazing Saddles had suggested taking the ferry back to the city, but after that burger I had a whole new set of calories to burn, so I headed back the way I’d come. It was pretty easy to follow the same route, though I do recommend a detour to the Dolores Park Cafe, which I checked out since its only a block from my new place. Their hot chocolate hit the spot, though you do need to be a fan of canines since its across the street from a great dog park.

If you find yourself in San Francisco with a bike and a few hours to spare, I highly recommend this ride. You’ll pass through some funky neighborhoods like Haight, two gorgeous parks and experience the Golden Gate bridge from a unique angle. It will also help justify all those lovely Bay Area meals that are tempting you…

Photo by Metrix X

How to not log personally-identifiable information – IP addresses are PII, so removing them from your server logs should be standard practice unless you have a specific need.

Inside Google's MapReduce infrastructure – Bloody hell, they're processing one exabyte of data a month! I didn't even know the term for 1,000 terabytes before, that's an astonishing number.

Netflix cloud storage – A white paper on Netflix's use of SimpleDB. I have to admit I've given up on it as a solution, the obstacles to large data loads overwhelmed me, but great to see they've had success.

Feedera – An intriguing take on 'personalized pagerank' for surfacing interesting Twitter articles. Had a great geek out last night with its creator Sachin Rekhi too.

Tealeaf – Remember that data entry field that cost Expedia $12m in lost sales? Julian Green had lots of similar tales to tell from his Ebay experiences, and apparently Tealeaf is a great tool for analyzing and diagnosing that sort of customer behavior.

I've decided to move to San Francisco. Much as I love Boulder, over the past year I've found so many of the people I'm working with or would like to work with are in the Bay Area, I'm making the leap.

Being the data-driven geek that I am, I wanted to understand the rental situation for different neighborhoods before I chose an apartment, so I watched Craigslist for a couple of weeks and collected 160 properties from around the city that met my basic criteria of having one bedroom and accepting a small dog. Once I had those in a CSV file, I could then upload them to OpenHeatMap to get this visualization:

http://www.openheatmap.com/view.html?map=PlectrumsColeridgeDeming

This actually helped me narrow down my search, since it made it clear that SoMa and the Embarcadero area were too pricey, and pointed me towards Mission and Lower Nob as my main targets. Over the next few days I'm going to be trying to pick out an apartment, and then moving right after I speak at Defrag next week.

If you like this sort of thing, you should also check out PadMapper, a Y-Combinator startup that has a great apartment search interface. I need to get them using OpenHeatMap though, I'd love to see their data in a non-map-pin form!

Photo by Brian Gurrola

How removing one data field saved $12m – A perfect example of why testing against reality is so important. It’s like code profiling, you never know where the bottlenecks are until you measure. via Felix Salmon

Ack – I’m doing a regular series over at ReadWriteWeb, trying to cover some of the folklore that’s typically passed down informally through the programmer ranks. I did one on using grep to search your source, and forgot to mention Ack, despite having used it and loved it with Textmate. Luckily the commenters kept me honest, so check it out, it really is better than grep for source code searching.

Google Refine – A very tasty looking open-source project from Google, designed to make it easy to import semi-structured data. Alex Dong has got my hopes up that it might be the missing tool for data scientists that I keep dreaming of, so I’ll be testing it out as soon as I have a chance.

GISCloud – The GIS world is one of the last holdouts where desktop software still rules. It’s inevitable that the tools will migrate to the cloud, so I’m excited to see how this progresses. via Andraz Tori

What Killed Aiyana Stanley-Jones? – This story is important because it builds a rounded picture of the factors that led to the death of a little girl in Detroit. Whatever your political persuasion, there will be parts that will back up your existing views, but other sections that may surprise you. With everything from reality TV crews to corrupt judges playing their part, there are no easy conclusions. It’s the sort of article that we need more of, exploring the problems rather than just cherry-picking them as evidence to push a solution.

Photo by Ian S

If I was the Lord High Dictator of English Usage, the very first term I'd ban is NoSQL, since all it does is enrage potential sympathizers whilst failing to accurately summarize what the new wave of database technologies have in common. Right after that though, I'd turn my beady eye towards 'data journalism', and give it a good hard stare.

The term actually has a long and illustrious history, if you consider it a synonym for database journalism. The trouble is, every modern journalist is using databases constantly, even if it's just via LexisNexis or Google searches. This makes the original term so broad to be meaningless.

I'd prefer to reserve the name for some of the interesting and unique work that's been emerging, work that's driven by a lot of the same shifts that have propelled Big Data into prominence. Here's the characteristics that I think true data journalism stories should possess:

– The data is a lead protagonist. It's common for trend stories to reach for a few statistics to back up a pre-determined conclusion, but this use of data as a Greek chorus is seldom very enlightening or rigorous, as it lends itself to cherry picking. I'm much more interested when the data is treated as an interview subject by the journalist, asked questions and the answers are allowed to drive the story's conclusion. With the Wikileaks dumps, the data is the lead character in most of the reports, and it's unearthed some unexpected results.

– The source material is public. If you quote a named source to back up your story, then anyone who wants to check whether you're distorting them can go back and talk to that person. With data-driven journalism, the only way to keep reporters honest and enable a real debate is to make the original information you base your conclusions on publicly available. Otherwise it's like using unnamed sources, you require a leap of faith from the reader that you're not being mislead. The Guardian is a shining example of how easy this can be, but the New York Times consistently refuses to release copies of the original source documents they base their stories on.

– There's real detective work involved. Is reporting on the unemployment rate or stock market data journalism? Almost always it's just repeating a pre-digested number, with a hand-wavey explanation throwing in for good measure – "The stock market was up today because of <random correlation>". What I love instead is when a reporter is clever about finding unusual data sources or powerful tools to uncover new information, often that was hidden in plain sight. My favorite recent example is Marshall Kirkpatrick's use of Needlebase to uncover information on Twitter's new data center, just by analyzing public Tweets from their employees.

On second thoughts, forget about changing the name. These principles are going to win out because they lead to more interesting and trustworthy stories, no matter what you call the genre. So let's just call it plain old good journalism instead.

Photo by Blake Patterson

I've been writing code almost as long as I can remember, and I've had a weird, reflective relationship with my projects, they regularly change my life. Here's the code that I built, and that helped build me.

28th April 198X – Anna's birthday card

My first taste of how powerful software could be was for my sister's birthday, when I was maybe eight or nine. I'd been fooling around with the wonderful programming manual that came with the ZX Spectrum, and found a listing that played 'Happy birthday to you' via BEEP commands. I combined that with some blinking ASCII graphics I built myself to approximate a birthday cake, and presented it to her and the rest of the family. They all told me they loved it, and that was a heady feeling. That joy in showing off something I've made myself continues to be the best part of my work, even if I've never managed to do anything that's quite as comprehensible to my family since.

1991 – Mars Landscape

When I was 15 I made my first money by writing code. I put together a procedural texture generator in BASIC for the Archimedes series of computers, and made 15 pounds when Acorn User published it! Sadly it would be another six years before I'd earn anything else through my software, so I had to rely on supermarket shelf-stacking instead.

1993 – Ped => Hex

Inspired by Coldcut's pioneering video work, and the 'Public Domain' disk-sharing-by-mail world that was the poor man's BBS, I put together a demo in ARM assembler. I was most proud of the 'ohhs' and 'ahhs' it pulled out of people, not because it was technically astounding but because I used every trick I could to make it have an impact, from strobing to a psychotropic color palette. In that sense it was a lot like a logical continuation of Anna's birthday card.

1994 – MUDs

Before there was World of Warcraft, there were text-based 'Multi-User Dungeons'. I'd got hooked on them a couple of year before, when me and a friend would sneak into the University of Cambridge's computer labs, but when I went away to college it became a debilitating addiction. I ended up getting an 18% overall mark averaged over all my first year exams and coursework, and got married, disasterously, to a girl I'd originally met through one of the systems when I was just 19. I spent time upgrading an maintaining some code for one the systems I used, which would have been an excellent foundation for the exploding world of the web, if I hadn't decided that TCP/IP networking wasn't interesting enough to keep working on!

1996 – XPacman

When I went away to university, my Acorn was becoming obsolete and I couldn't afford to get another computer, so all of my programming was on Manchester's set of Sun terminals. I grew to abhore the XWindows network-based model of GUI development, but I did learn enough to create a Pacman clone. I loved the calm and quiet of the computer labs, as a contrast to an increasingly stormy home life.

1997 – Diablo for the Playstation

After I graduated with my BSc I was giddy at the prospect of being able to work full-time writing software, and especially to follow my dream of writing game software. I got my first job with Climax Industries (endless comedy value) working on porting Diablo from the PC to the Playstation 1. The salary was $15,000 a year, which after measuring my supermarket wages hourly sounded like a mind-blowing amount of money. I soon learned I was actually making less than I did stacking shelves, but the experience of working with some very savvy engineers and learning to read other people's code was a crucial education. Gosport was by far the roughest place I've ever lived, I got mugged before I'd been there a week, and the paltry pay, high rent and an unemployed wife drove me deeper into debt. I was at the point of getting letters threatening county court judgments, and that experience has left me deeply averse to ever taking out any sort of credit. After six months I found a better job up in Scotland, and could begin to dig myself out of that hole.

2001 – Pete's Plugins

My marriage finally ended, and I decided to take some time out to travel. My first stop was the US, where I intended to work for maybe six months and see a bit of the country. I fell in love, both with the country and a girl, and I've been here ever since. I'd been learning my craft in the game industry for the last few years, but it wasn't the creative endeavour I'd dreamed of as a kid. As an outlet for that, I started writing software that harked back to my demo days, effects that I could project onto screens at clubs and concerts, with me controlling the video in a way that complemented the music. I loved being able to actually see my audience reacting to my work, right in front of me, that's so rare in coding, it drove me to produce more than I'd ever imagined I could. Over the course of a year, I wrote over forty effects in my spare time, and released them as open-source plugins for all of the popular VJ packages. Users kept bugging me to port them over to something called After Effects, which I'd heard was a tool professional video producers used. Once I did, the reaction was amazing, people were literally emailing me to ask how they could pay me for the effects. They were used to buying three or four effects for a thousand dollars, and I was giving them forty for free. I was slowly catching on to the whole capitalism thing, so I quit my day job to produce a new set of professional plugins for that market.

2003 – Apple Motion

It rapidly became obvious that setting up a small business, even one with customers clamoring to give me money, was not going to be possible as an immigrant. Happily Apple approached me on the basis of my open source work, offered to buy out the few months of work I'd put into my business and have me help build a new secret project they were working on. I spent five years working deeply with all sorts of graphics technologies within Apple, using my game engine experience to help deal with graphics driver issues and all sorts of GPU programming adventures.

2006 – SearchMash

I was growing increasingly fascinated by the power of the web. I could already see the writing on the wall for desktop software, and knew I wanted to work with more creative control, which meant becoming my own boss. SearchMash was my first real experiment on the web, a klunky Java applet that displayed a split-screen view of Google's search results. I went on to follow similar threads with GoogleHotKeys, and then dived into applying my graphics skills at analyzing large data sets with more social tools like Event Connector. I'm not even sure how many of these experiments are still functioning, but it was a real education in what actually works for engaging people.

2008 – Mailana

My green card finally came through, so I left Apple and set out to build some of the tools I wished I'd have while I was there, a service to locate experts and external contacts within large companies using only the information lying around on their Exchange servers. The experience was a fantastic education in how not to kick off a startup. Though I did run a few small pilots, I focused almost entirely on the technology for a year. I talked to potential investors more than I talked to potential customers. Inevitably I realized after a year that I had a big bag of technology but no business. The only bright side was that I'd ended up getting some love when, on a whim, I applied that processing pipeline to Twitter

2009 – FanPageAnalytics

After that realization, I spent some time metaphorically wandering in the wilderness. I knew the enterprise approach was dead, but I couldn't get a good handle on a consumer angle. I moved to Boulder to get some mentoring at Techstars, but none of my experiments worked. The most interest I had was for a Gmail-based service that would send useful information to you about people who'd emailed you, the same sort of thing you get when you Google someone's name. I couldn't find an API that would allow me to do that sort of query, so I decided to write my own search engine to crawl the public pages of the sites I cared about. While I was doing that, I noticed that Facebook had lots of interesting information in those crawlable pages, and thought there might be a business in providing analytics for brands, giving them extra information about their (and their competitor's) fan pages on Facebook. FanPageAnalytics was born, and then quickly died as I got entangled with Facebook's legal department.

2010 – OpenHeatMap

I had a lot of time to think about what was working, and what wasn't, as I waited for the Facebook saga to end. I'd had over 500,000 people visit my Five Nations of Facebook map in just a few days, but other than the extra readership I didn't build a business relationship with any of them. I realized that people loved these online maps, that this was an amazing distribution channel for reaching an audience, and I wanted to build more, and help other people build their own. I couldn't find existing tools that did what I wanted, so OpenHeatMap was born, and that's where I am right now. I'm happy with the progress the free site is making, and a revenue-generating spin-off is in early testing. I wonder how this one will change my life?