Biking over the Golden Gate Bridge – Downtown San Francisco to Sausalito

Since my apartment hunting was over more quickly than I expected, and I’ve been enjoying the San Francisco food far too much, so I decided to go for a bike ride this morning. With the Golden Gate bridge looming, how could I resist? The guys at Blazing Saddles gave me some general directions when I rented my bike, but there were a few sections that were tricky to follow, so I’ve mapped the route I ended up taking from downtown San Francisco, and give my own directions below. I rode from the city to Sausalito and back again, with a nice lunch spot at the turn-around point, and it was about 22 miles total with some non-trivial hills.

For convenience I’ve started the map at Powell Street BART station, but I actually did a more complex route through downtown, since I had errands to run. There’s a great bike lane down Market Street though, so I recommend taking that south until you get to Page Street. This is then a straight shot to Golden Gate park, and it’s mostly quiet, residential and bike friendly, though you will need to navigate frequent stop signs. In the park itself you take bike paths through the north-east corner to Aguello Boulevard. That heads north towards the massive Presidio park, and it can be a little tough dealing with the traffic despite the bike lane. Once in the park, continue along Aguello until Washington splits off to the left. Washington will take you up and over the ridge, and then merges into Lincoln. A little way down Lincoln, turn left on Merchant, and follow that until you’re approaching the PCH/101.

This was the hardest part of the ride for me to follow. I knew that on weekends, the western sidewalk of the bridge was reserved for cyclists, but I had a hell of a time figuring out how to get there. Eventually I worked out you had to take the tunnel under the roadway from Merchant, then bike up on the east side past the gift shop, then follow another tunnelled path under the roadway back to the western side and the bridge’s sidewalk.

The bridge itself was fantastic to bike on, even on a busy Sunday morning. The weather was clear and sunny with breath-taking views across the water, the grade was steady and even the most aggressive bikers had plenty of room to swerve around tourists. At the other end you come off into a parking lot, and can then head onto Alexander Avenue to Sausalito. This can be a bit hairy, since the road is narrow and steep in sections, but the drivers seemed quite bike-aware thankfully. In Sausalito itself, you end up riding along the waterfront. There’s a lot of choices for refreshments, but I was intrigued by a tiny hole-in-the-wall place that simply promised ‘Hamburgers’:

Hamburgers

Yelp gave it a thumbs-up, so I ventured in for a cheeseburger and fries. Man, that tasted good! They’re apparently a bit Soup-Nazi in their customer service, but I didn’t see anything other than slight surliness. It was lucky I got there early because they soon had a queue out the door, so I guess I’m not the only one to be hooked by their food. There’s almost no space to eat inside the restaurant, but the park across the road had a gorgeous view over the bay and plenty of benches.

Blazing Saddles had suggested taking the ferry back to the city, but after that burger I had a whole new set of calories to burn, so I headed back the way I’d come. It was pretty easy to follow the same route, though I do recommend a detour to the Dolores Park Cafe, which I checked out since its only a block from my new place. Their hot chocolate hit the spot, though you do need to be a fan of canines since its across the street from a great dog park.

If you find yourself in San Francisco with a bike and a few hours to spare, I highly recommend this ride. You’ll pass through some funky neighborhoods like Haight, two gorgeous parks and experience the Golden Gate bridge from a unique angle. It will also help justify all those lovely Bay Area meals that are tempting you…

Five short links

Fiveoclock
Photo by Metrix X

How to not log personally-identifiable information – IP addresses are PII, so removing them from your server logs should be standard practice unless you have a specific need.

Inside Google's MapReduce infrastructure – Bloody hell, they're processing one exabyte of data a month! I didn't even know the term for 1,000 terabytes before, that's an astonishing number.

Netflix cloud storage – A white paper on Netflix's use of SimpleDB. I have to admit I've given up on it as a solution, the obstacles to large data loads overwhelmed me, but great to see they've had success.

Feedera – An intriguing take on 'personalized pagerank' for surfacing interesting Twitter articles. Had a great geek out last night with its creator Sachin Rekhi too.

Tealeaf – Remember that data entry field that cost Expedia $12m in lost sales? Julian Green had lots of similar tales to tell from his Ebay experiences, and apparently Tealeaf is a great tool for analyzing and diagnosing that sort of customer behavior.

Mapping apartment prices in San Francisco

Sfapartmentmap

I've decided to move to San Francisco. Much as I love Boulder, over the past year I've found so many of the people I'm working with or would like to work with are in the Bay Area, I'm making the leap.

Being the data-driven geek that I am, I wanted to understand the rental situation for different neighborhoods before I chose an apartment, so I watched Craigslist for a couple of weeks and collected 160 properties from around the city that met my basic criteria of having one bedroom and accepting a small dog. Once I had those in a CSV file, I could then upload them to OpenHeatMap to get this visualization:

http://www.openheatmap.com/view.html?map=PlectrumsColeridgeDeming

This actually helped me narrow down my search, since it made it clear that SoMa and the Embarcadero area were too pricey, and pointed me towards Mission and Lower Nob as my main targets. Over the next few days I'm going to be trying to pick out an apartment, and then moving right after I speak at Defrag next week.

If you like this sort of thing, you should also check out PadMapper, a Y-Combinator startup that has a great apartment search interface. I need to get them using OpenHeatMap though, I'd love to see their data in a non-map-pin form!

Five short links

I5
Photo by Brian Gurrola

How removing one data field saved $12m – A perfect example of why testing against reality is so important. It’s like code profiling, you never know where the bottlenecks are until you measure. via Felix Salmon

Ack – I’m doing a regular series over at ReadWriteWeb, trying to cover some of the folklore that’s typically passed down informally through the programmer ranks. I did one on using grep to search your source, and forgot to mention Ack, despite having used it and loved it with Textmate. Luckily the commenters kept me honest, so check it out, it really is better than grep for source code searching.

Google Refine – A very tasty looking open-source project from Google, designed to make it easy to import semi-structured data. Alex Dong has got my hopes up that it might be the missing tool for data scientists that I keep dreaming of, so I’ll be testing it out as soon as I have a chance.

GISCloud – The GIS world is one of the last holdouts where desktop software still rules. It’s inevitable that the tools will migrate to the cloud, so I’m excited to see how this progresses. via Andraz Tori

What Killed Aiyana Stanley-Jones? – This story is important because it builds a rounded picture of the factors that led to the death of a little girl in Detroit. Whatever your political persuasion, there will be parts that will back up your existing views, but other sections that may surprise you. With everything from reality TV crews to corrupt judges playing their part, there are no easy conclusions. It’s the sort of article that we need more of, exploring the problems rather than just cherry-picking them as evidence to push a solution.

What is data journalism?

Usbpunchedcard
Photo by Ian S

If I was the Lord High Dictator of English Usage, the very first term I'd ban is NoSQL, since all it does is enrage potential sympathizers whilst failing to accurately summarize what the new wave of database technologies have in common. Right after that though, I'd turn my beady eye towards 'data journalism', and give it a good hard stare.

The term actually has a long and illustrious history, if you consider it a synonym for database journalism. The trouble is, every modern journalist is using databases constantly, even if it's just via LexisNexis or Google searches. This makes the original term so broad to be meaningless.

I'd prefer to reserve the name for some of the interesting and unique work that's been emerging, work that's driven by a lot of the same shifts that have propelled Big Data into prominence. Here's the characteristics that I think true data journalism stories should possess:

The data is a lead protagonist. It's common for trend stories to reach for a few statistics to back up a pre-determined conclusion, but this use of data as a Greek chorus is seldom very enlightening or rigorous, as it lends itself to cherry picking. I'm much more interested when the data is treated as an interview subject by the journalist, asked questions and the answers are allowed to drive the story's conclusion. With the Wikileaks dumps, the data is the lead character in most of the reports, and it's unearthed some unexpected results.

The source material is public. If you quote a named source to back up your story, then anyone who wants to check whether you're distorting them can go back and talk to that person. With data-driven journalism, the only way to keep reporters honest and enable a real debate is to make the original information you base your conclusions on publicly available. Otherwise it's like using unnamed sources, you require a leap of faith from the reader that you're not being mislead. The Guardian is a shining example of how easy this can be, but the New York Times consistently refuses to release copies of the original source documents they base their stories on.

There's real detective work involved. Is reporting on the unemployment rate or stock market data journalism? Almost always it's just repeating a pre-digested number, with a hand-wavey explanation throwing in for good measure – "The stock market was up today because of <random correlation>". What I love instead is when a reporter is clever about finding unusual data sources or powerful tools to uncover new information, often that was hidden in plain sight. My favorite recent example is Marshall Kirkpatrick's use of Needlebase to uncover information on Twitter's new data center, just by analyzing public Tweets from their employees.

On second thoughts, forget about changing the name. These principles are going to win out because they lead to more interesting and trustworthy stories, no matter what you call the genre. So let's just call it plain old good journalism instead.

My life in software

Floppies
Photo by Blake Patterson

I've been writing code almost as long as I can remember, and I've had a weird, reflective relationship with my projects, they regularly change my life. Here's the code that I built, and that helped build me.

28th April 198X – Anna's birthday card

My first taste of how powerful software could be was for my sister's birthday, when I was maybe eight or nine. I'd been fooling around with the wonderful programming manual that came with the ZX Spectrum, and found a listing that played 'Happy birthday to you' via BEEP commands. I combined that with some blinking ASCII graphics I built myself to approximate a birthday cake, and presented it to her and the rest of the family. They all told me they loved it, and that was a heady feeling. That joy in showing off something I've made myself continues to be the best part of my work, even if I've never managed to do anything that's quite as comprehensible to my family since.

1991 – Mars Landscape

When I was 15 I made my first money by writing code. I put together a procedural texture generator in BASIC for the Archimedes series of computers, and made 15 pounds when Acorn User published it! Sadly it would be another six years before I'd earn anything else through my software, so I had to rely on supermarket shelf-stacking instead.

1993 – Ped => Hex

Inspired by Coldcut's pioneering video work, and the 'Public Domain' disk-sharing-by-mail world that was the poor man's BBS, I put together a demo in ARM assembler. I was most proud of the 'ohhs' and 'ahhs' it pulled out of people, not because it was technically astounding but because I used every trick I could to make it have an impact, from strobing to a psychotropic color palette. In that sense it was a lot like a logical continuation of Anna's birthday card.

1994 – MUDs

Before there was World of Warcraft, there were text-based 'Multi-User Dungeons'. I'd got hooked on them a couple of year before, when me and a friend would sneak into the University of Cambridge's computer labs, but when I went away to college it became a debilitating addiction. I ended up getting an 18% overall mark averaged over all my first year exams and coursework, and got married, disasterously, to a girl I'd originally met through one of the systems when I was just 19. I spent time upgrading an maintaining some code for one the systems I used, which would have been an excellent foundation for the exploding world of the web, if I hadn't decided that TCP/IP networking wasn't interesting enough to keep working on!

1996 – XPacman

When I went away to university, my Acorn was becoming obsolete and I couldn't afford to get another computer, so all of my programming was on Manchester's set of Sun terminals. I grew to abhore the XWindows network-based model of GUI development, but I did learn enough to create a Pacman clone. I loved the calm and quiet of the computer labs, as a contrast to an increasingly stormy home life.

1997 – Diablo for the Playstation

After I graduated with my BSc I was giddy at the prospect of being able to work full-time writing software, and especially to follow my dream of writing game software. I got my first job with Climax Industries (endless comedy value) working on porting Diablo from the PC to the Playstation 1. The salary was $15,000 a year, which after measuring my supermarket wages hourly sounded like a mind-blowing amount of money. I soon learned I was actually making less than I did stacking shelves, but the experience of working with some very savvy engineers and learning to read other people's code was a crucial education. Gosport was by far the roughest place I've ever lived, I got mugged before I'd been there a week, and the paltry pay, high rent and an unemployed wife drove me deeper into debt. I was at the point of getting letters threatening county court judgments, and that experience has left me deeply averse to ever taking out any sort of credit. After six months I found a better job up in Scotland, and could begin to dig myself out of that hole.

2001 – Pete's Plugins

My marriage finally ended, and I decided to take some time out to travel. My first stop was the US, where I intended to work for maybe six months and see a bit of the country. I fell in love, both with the country and a girl, and I've been here ever since. I'd been learning my craft in the game industry for the last few years, but it wasn't the creative endeavour I'd dreamed of as a kid. As an outlet for that, I started writing software that harked back to my demo days, effects that I could project onto screens at clubs and concerts, with me controlling the video in a way that complemented the music. I loved being able to actually see my audience reacting to my work, right in front of me, that's so rare in coding, it drove me to produce more than I'd ever imagined I could. Over the course of a year, I wrote over forty effects in my spare time, and released them as open-source plugins for all of the popular VJ packages. Users kept bugging me to port them over to something called After Effects, which I'd heard was a tool professional video producers used. Once I did, the reaction was amazing, people were literally emailing me to ask how they could pay me for the effects. They were used to buying three or four effects for a thousand dollars, and I was giving them forty for free. I was slowly catching on to the whole capitalism thing, so I quit my day job to produce a new set of professional plugins for that market.

2003 – Apple Motion

It rapidly became obvious that setting up a small business, even one with customers clamoring to give me money, was not going to be possible as an immigrant. Happily Apple approached me on the basis of my open source work, offered to buy out the few months of work I'd put into my business and have me help build a new secret project they were working on. I spent five years working deeply with all sorts of graphics technologies within Apple, using my game engine experience to help deal with graphics driver issues and all sorts of GPU programming adventures.

2006 – SearchMash

I was growing increasingly fascinated by the power of the web. I could already see the writing on the wall for desktop software, and knew I wanted to work with more creative control, which meant becoming my own boss. SearchMash was my first real experiment on the web, a klunky Java applet that displayed a split-screen view of Google's search results. I went on to follow similar threads with GoogleHotKeys, and then dived into applying my graphics skills at analyzing large data sets with more social tools like Event Connector. I'm not even sure how many of these experiments are still functioning, but it was a real education in what actually works for engaging people.

2008 – Mailana

My green card finally came through, so I left Apple and set out to build some of the tools I wished I'd have while I was there, a service to locate experts and external contacts within large companies using only the information lying around on their Exchange servers. The experience was a fantastic education in how not to kick off a startup. Though I did run a few small pilots, I focused almost entirely on the technology for a year. I talked to potential investors more than I talked to potential customers. Inevitably I realized after a year that I had a big bag of technology but no business. The only bright side was that I'd ended up getting some love when, on a whim, I applied that processing pipeline to Twitter

2009 – FanPageAnalytics

After that realization, I spent some time metaphorically wandering in the wilderness. I knew the enterprise approach was dead, but I couldn't get a good handle on a consumer angle. I moved to Boulder to get some mentoring at Techstars, but none of my experiments worked. The most interest I had was for a Gmail-based service that would send useful information to you about people who'd emailed you, the same sort of thing you get when you Google someone's name. I couldn't find an API that would allow me to do that sort of query, so I decided to write my own search engine to crawl the public pages of the sites I cared about. While I was doing that, I noticed that Facebook had lots of interesting information in those crawlable pages, and thought there might be a business in providing analytics for brands, giving them extra information about their (and their competitor's) fan pages on Facebook. FanPageAnalytics was born, and then quickly died as I got entangled with Facebook's legal department.

2010 – OpenHeatMap

I had a lot of time to think about what was working, and what wasn't, as I waited for the Facebook saga to end. I'd had over 500,000 people visit my Five Nations of Facebook map in just a few days, but other than the extra readership I didn't build a business relationship with any of them. I realized that people loved these online maps, that this was an amazing distribution channel for reaching an audience, and I wanted to build more, and help other people build their own. I couldn't find existing tools that did what I wanted, so OpenHeatMap was born, and that's where I am right now. I'm happy with the progress the free site is making, and a revenue-generating spin-off is in early testing. I wonder how this one will change my life?

Five short links

Fivefingers
Photo by Jane Rahman

What do prototypes prototype? – In my last post I explored some of the problems with our model of prototypes, which inspired Joe Parry to point me at this in-depth paper. Very thought-provoking ideas on there being three axes to classify prototypes by; implementation, look and feel, and role.

How telephone directories transformed America – A hundred years before Facebook, our social structure was transformed by a book that listed our names and addresses, and gave strangers a way to contact us directly. I've often used the phone book analogy when discussing privacy, but I'd never really understood what a major shift it actually represented.

We ten million – That's a guesstimate of the number of aspiring novelists out there. Whenever I get daunted by the difficulty of making it with a startup, I remember all the amazing artists and writers I know who'd kill to share the odds we have. My favorite advice for success is "I was neither the most talented nor the most clever writer in my writing group, but I was the one who stuck with it." I hope that works for founders too, since my only super-power is the iron-plated stubborness I inherited from my grandmother.

Some cool things I heard in New Zealand – Sounds like a fascinating conference, gathering together 300 CEOs from the design world. Lots of great quotes, but the most insightful was "When asked why Method keeps innovating, he answered 'our people give a shit.'". I know exactly what he means, you can tell within a few minutes of dealing with most companies whether the people there actually care, or if they're so beaten down they're just picking up paychecks.

Exercise more to hack better – I smoked and barely exercised until I was 25 and moved to the US, but starting made an amazing difference in my own productivity, and in my general quality of life. It's funny, but I'd always assumed exercise made you tired. In fact that endorphin cocktail has been so helpful I try to get in a decent workout every morning, just to keep me on the ball with my work.