I was lucky enough to spend a few hours today with my friend Kevin Gates, one of the creators of Google's internal business intelligence systems, and it turned out to be a very thought-provoking chat. His mind was somewhat boggled that we were so data-obsessed at such an early stage in our life. Most people running analytics work at a large company and have a big stream of users to run experiments on. Our sample sizes are much smaller, which makes even conceptually simple approaches like A/B tests problematic. Just waiting long enough to get a statistically-significant results becomes a big bottleneck.

We've found ways around a lot of the technical issues, for example focusing on pre/post testing rather than A/B to speed up the process, but there's a bigger philosophical question. Is it even worth focusing on data when you only have tens of thousands of users?

The key for us is that we're using the information we get primarily for decision-making (should we build out feature X?) rather than optimization (how can we improve feature X?). Our quest is to understand what users are doing and what they want. Everything we're looking at should be actionable, should answer a product question we're wrestling with. To help answer that, I sketched out a diagram of how the information flows through our tools to the team:

The silhouettes show where people are looking at the results of our data crunching. The primary things that everyone on our team religiously watches are the daily report emails, and the UserTesting.com videos that show ordinary people using new features of our app. The daily reports are built on top of our analytics database, which is a Postgres machine with a homebrewed web UI to create, store, and regularly run reports on the event logs it holds. We built this when our requirements expanded beyond KissMetrics more funnel-focused UI, but we still use their web interface for some of our needs. Qualaroo is an awesome offshoot of KissMetrics that we use for in-app surveys, and we also refer to MailChimp's Mandrill dashboard and Urban Airship's statistics to understand how well our emails and push notifications are working. We have to use AppAnnie to keep track of our iOS download numbers and reviews over time.

We also have about twenty key statistics that we automatically add to a 'State of the App' Google Docs spreadsheet every day. This isn't something we constantly refer to, but it is crucial when we want to understand trends over weeks or months.

Over the last 18 months we've experimented with a lot of different approaches and sources of data, but these are the ones that have proved their worth in practice. It doesn't look the same as a large company approach to analytics, but this flow has been incredibly useful in our startup environment. It has helped us to make better and faster decisions, and most importantly spot opportunities we'd never have seen otherwise. If you're a small company and are feel like you're too early to start on analytics, you may be surprised by how easy it is to get started and how much you get out of it. Give simple services like KissMetrics a try, and I bet you'll end up hooked!

Photo by Oatsy 40

My last post was a quick rant about the need for a decent open geocoder, but what's wrong with the ones we have? I've created a command-line tool to explore their quality: https://github.com/petewarden/geocodetest.

As a first pass, I pulled together a list of six addresses, some from my past and a few from spreadsheets users have uploaded to OpenHeatMap. The tool runs through the list (or any file of addresses you give it) and geocodes them through DSTK and Nominatim, returning a CSV of whether the locations are within 100m of Google's result. Run the script with -h to see all the options.  Here are the results, produced by running ./geocodetest.rb -i testinput.txt

dstk,google,nominatim,address
Y,Y,Y,2543 Graystone Place, Simi Valley, CA 93065
Y,Y,N,400 Duboce Ave, #208, San Francisco CA 94117
Y,Y,N,11 Meadow Lane, Over, Cambridge, CB24 5NF, UK
N,Y,N,VIC 3184, Australia
N,Y,Y,Lindsay Crescent, Cape Town, South Africa
N,Y,N,3875 wilshire blvd	los angeles	CA

The first three are standard test cases for me, so it's not be a massive surprise that my DSTK (based on Schuyler Erle and GeoIQ's original work) works better than Nominatim for two of them. It does highlight one of the reasons I've struggled to use Nominatim though – it's not good at coping with alternative address forms. This makes it quite brittle, especially around addresses like the UK where there are multiple common permutations of village, city, and county names. Nominatim doesn't return any results for #2 or #3 at all, when I'd hope for at least a town-level approximation.

The Australian postal code is about 30 km from Google's result, whereas the open GeoNames data in the DSTK gets me to within 400m of Google. Nominatim does much better on the SA address, since I haven't imported OSM data into the DSTK for anywhere but the UK. I did have to correct the original user-entered spelling of 'Cresent' first though, and I'd love to see an open geocoder that was robust to this sort of common mistake. The last address is another sloppy one, but we should be able to cope with that one too!

Part of the reason there hasn't been more progress on open geocoders is that the problems are not very visible. I hope having an easy test harness changes that, and while this first pass is far from scientific, it's already inspired me to put in several fixes to my own code. I'm a big fan of the effort that's been put into the Nominatim project (I'm using their OSM loading code myself) I'm just disappointed that the results haven't been good enough to build services like OpenHeatMap on top of. I'll be expanding this tool to cover more addresses and so build a better 'map' of how we're doing, and what remains to be done. I'm excited by the opportunities to make progress here, I'll be busy working more on my own efforts and I can't wait to hear other folks thoughts too.

Photo by Werner Kunz

A few years ago I had what I thought was a simple problem. I had a bunch of place names, and I needed to turn them into latitude and longitude coordinates. To my surprise, it turned out to be extremely hard. Google has an excellent geocoder, but you're only allowed to use it for data you're displaying on Google maps, and there are rate limits and charges if you use it in bulk. Yahoo has an excellent array of geo APIs with much better conditions, but there are still rate limits and their future was in doubt even then!

So, I ended up hacking up my own very basic solution based on open data. It turned out to be a fascinating problem, one you could spend a lifetime on, trying to draw a usable, detailed picture of the world from freely available data. I bulked up the underlying data and algorithms, and it became the core of the Data Science Toolkit. Turning addresses into coordinates may sound like a strange obsession, but it has become my white whale.

There are some folks who agree that this is an important problem, but I've been surprised there aren't more. Placenames describe our world, and we need an open and democratic way for machines to interpret them. Almost any application that uses locations needs to do this operation, and right now we have no alternative to commercial systems.

What are the practical impacts of this? We've got no control over what our neighborhoods are called, or how they're defined. We can't fix problems in the data that impact us, like correcting the location of our address so that delivery drivers can find us. We can't build applications that take in large amounts of address data unless we can afford high fees, which cuts out a whole bunch of interesting projects.

This is on my mind because I'm making another attack on improving the DSTK solution. I've already added a lot of international postal codes thanks to GeoNames, but next I want to combine the public domain SimpleGeo point-of-interest dump with OpenStreetMap data to see if I can synthesize more addressable ranges for at least some more countries. That will be an interesting challenge, but if I get some usable it opens the door to adding more coverage through any open data set that combines street addresses and coordinates. I can't wait to see where this takes me!

Photo by Alan Levine

You can't sacrifice partition tolerance – A convincingly (and amusingly) argued case that you can never trade off the P in the CAP theorem.

Topic discovery with Apache Pig and Mallet – This sort of thing used to be magic, now you can assemble it from off-the-shelf components.

The insanely confusing path to legal immigration – I've almost made it through my immigration story, knock on wood I'll be taking my citizenship oath in May after twelve years, but it's been tough to explain to my American friends quite how convoluted the process is. This chart will help!

dancer.js – The web continues to devour the software world. Javascript can now handle both the fast rendering and the audio analysis you need for this music-responsive visualization.

Click dataset – Over 50 billion real-world HTTP requests. I'm certain there are identifiable elements in this data, but I think Arvind's right that researchers have proved this so convincingly that they won't bother to highlight them, and malicious users will never talk about it, so for some approximation of matters, it doesn't matter.

Picture by Jacob Haas

I have fallen in love with Vagrant over the last year, it turns an entire logical computer as a single unit of software. In simple terms, you can easily set up, run, and maintain a virtual machine image with all the frameworks and data dependencies pre-installed. You can wipe it, copy it to a different system, branch it to run experimental changes, keep multiple versions around, easily share it with other people, and quickly deploy multiple copies when you need to scale up. It's as revolutionary as the introduction of distributed source control systems, you're suddenly free to innovate because mistakes can be painlessly rolled back, and you can collaborate other people without worrying that anything will be overwritten.

Before I discovered Vagrant, I'd attempted to do something similar with my Data Science Toolkit package, distributing a VMware image of a full linux system with all the software and data it required pre-installed. It was a large download, and a lot of people used it, but the setup took more work than I liked. Vagrant solved a lot of the usability problems around downloading VMs, so I've been eager to create a compatible version of the DSTK image. I finally had a chance to get that working over the weekend, so you can create your own local geocoding server just by running:

vagrant box add dstk http://static.datasciencetoolkit.org/dstk_0.41.box

vagrant init

The box itself is almost 5GB with all the address data, so the download may take a while. Once it's done go to http://localhost:8080 and you'll see the web interface to the geocoding and unstructured data parsing functions.

I've updated the US address data using the most recent Census data from 2012, rebuilt the system around Ubuntu 12.04, and incorporated a lot of virtual memory setting changes that improve the stability of the system when it's dealing with large volumes of API calls. I've released an EC2 AMI with all these changes too, and the full instructions for setting up your own server are at http://www.datasciencetoolkit.org/developerdocs#amazon.

Photo by Mike P

For most of my life I've managed to avoid being a boss. Since I helped start Jetpac the responsibility has crept up on me, and not coincidentally so have a few grey hairs. I'm still loving my job, but managing is tough, but not always in the ways I expected.

One of my biggest surprises is how bad I was at leading a team of engineers. I'd spent a long time as a senior guy on various decent-sized teams, taking a lot of initiative and making a lot of decisions, so I thought leading would be a big but incremental step. Instead, I've actually had to unlearn a lot of what I'd picked up over the last 15 years.

In particular, I found that my enjoyment of the debate about ways to implement features went from being valuable to toxic. As a team contributor, I was used to chiming in while we all hashed out the right approach, matching wits and learning in a back-and-forth debate. For the first few months here I did the same thing, and was mystified that something just didn't seem right. The discussions seemed more stilted, and it never felt like everyone had truly bought in to the conclusions. People weren't as enthusiastic as I'd expect, and problems that we should have caught in the planning stages only became apparent much later.

It took me a while, but I finally realized I was in a different position. When I gave my opinion it carried more weight, and if I jumped in on every interesting detail I'd end up cutting the discussions short. That meant I never benefitted from the experience of the super-smart engineers I'm lucky enough to work with. I realized I have a different role, and I have to have a much lighter touch.

Instead of getting deeply involved in the implementation approaches, I've found it's worked much better to focus on the end-user goals of what we're trying to do, and communicating those to the engineering team. An important part of that is asking questions about how they think different approaches will meet particular goals. "Do you think this will get more people exploring this feature?". "Will that get more people entering recommendations?". The key difference from my previous approach is that I give them ownership of the way they reach those goals. As long as they're able to meet them, I don't care how they get there! The team take pride in their work, so I've never had to worry about code quality, and the end-user results have been amazing. We've ended up with some very successful innovations that I'd never have dreamt up in a million years!

If you're helping lead a team, think about what you truly care about. I bet it's outcomes you want, and you'll need to step back from your own preferences on the details if you want a team of creative people to achieve them. Point them at the right mountain, make sure you're giving them good crampons, maps, and guide books, but let them pick a route up themselves!

Photo by Alex Loach

Passing data from server to Javascript on page load – A strong treatment of a grubby little subject that anyone who writes a non-trivial web app has to think about. We have a much more ad hoc version of this, and I'd probably stick to a whitelist of known operations rather than passing in a function name as a string, but I like the approach.

Vaurien, the Chaos TCP proxy – I'm itching to use this, without any pressing justification. There's just something very appealing about throwing glitches and noise into any system and seeing what happens.

How food shapes our cities – This gave me a sense of wonder at how far we've come so quickly, with just a couple of centuries (or these days a few hundred miles and a border or two) separating us from desperately unreliable food supplies.

dstk_excel – Despite its issues, I love github's new search, it helped me discover this Excel interface to my Data Science Toolkit! I love people sometimes.

Heather Arthur – And then sometimes people suck. It's actually good to see this get some attention, being respectful about other people's didn't come naturally to me. It took a good first lead programmer to point out that while I was being snobbish about the original Diablo code I was working on porting, the original engineers were rolling in money like Scrooge McDuck, so who was the idiot?

Photo by Nick J Webb

There are lots of advantages to developing for iOS devices in Javascript, either as a mobile website or through a native app that hosts a UIWebView. Debuggability is definitely not one of them though! You'll find yourself flying blind when you need to track down errors, especially compared to the awesome state of browser debuggers. There are techniques that can help though, so I wanted to give a quick overview of what we've ended up doing for Jetpac.

Local logging

If you're targeting Mobile Safari, it's comparatively easy to see your error messages when you're debugging, just enable it in the settings. It gets tricky with a UIWebView though, and we ended up using this custom URL scheme hack (which requires some native code changes) to get log messages appearing in the device console. It's also worth knowing that you can view the console even when you didn't run the app through the debugger (for example if you've installed it through the app store) by plugging in and looking in Organizer->Devices. You can even buy apps to let you view the console natively, which should make you think twice about putting any private information you don't want other apps to access in log messages!

Web inspector

You should check out the new iOS 6 remote debugger, which works with both Safari and UIWebView code. It's been extremely useful for digging into CSS issues, and saved our bacon when tracking down some weird script loading problems.

Catching errors in the wild

The most challenging part is getting information on problems that are happening to users with the released app. If you can't reproduce the issue locally with a device plugged in, how can you tell what went wrong?

The first step is attaching a callback to window.onerror(), which will be called whenever there's an uncaught exception. In iOS 5, you only get the error message, not the file or line, and for various reasons we've had to minify and inline code anyway, so iOS 5's addition of the line number and file name isn't very helpful. What we really need is the call stack, which just doesn't get returned in any form on Mobile Safari.

Scarily, Javascript is such a flexible language that it's possible to do a crazy level of modification of the function calling internals, enough to write user-level tracing for every function! I actually got a version of this Function.prototype hack partially running as an experiment, but the breadth of it scared me. I also realized that I didn't need every function in the call stack, I mostly just wanted to know what part of my code had triggered the problem. What I ended up doing was manually wrapping functions I know about, and outputting information about them in onerror(). It's still an extremely hacky hack, but it's been very useful as we've been tracking down tough release problems, so here's the code I ended up using:

This won't run out of the box, but it should give you an idea of what we're doing. As part of our server-side code we have an error-reporting endpoint that we post the details of any release errors to, /jserror, and that sends on an email to the team.

The heavy lifting happens in the wrapFunctions() call, which replaces each function in an object with a wrapper that first calls the supplied 'before' function (in our case just pushing onto the callstack), then the original function, followed by 'after'. There are no guarantees about the correctness of the code in all cases, the prototype stuff especially scares me, but it has worked in practice on our code base.

I tend to use this pretty sparingly to wrap our own code, rather than jQuery or other frameworks, since most of the errors are in our functions, and I'm worried about sprinkling too much voodoo over our code base. Despite those caveats, it's been a massive help in tracking down our issues.

Photo by Trey Ratcliff

A while ago I was having drinks with a Google employee, and we started discussing privacy problems. He asked me why Buzz had received so much bad press for its email analysis when Facebook and other social networks had been doing the same thing for years? He also pressed me on why the iPhone tracking story had become such a big issue.

People have a mental model of what devices and services are for, and get freaked out when someone changes the rules. Nobody understands constantly-changing space-shuttle-control-panel privacy settings within services, but everyone knows that LinkedIn is for business relationships, and Facebook is for friends. Users try to protect their privacy by limiting information to sites that serve the audiences they want it to reach.

When Google changed from an email and search provider to a service that could broadcast semi-public updates to her friends, it became unclear where information she'd previously shared would end up. When Apple switched from a phone and computer builder to something that followed your movements, that crossing of boundaries was the real problem. Nobody would have blinked an eye at the idea of a Garmin device keeping a file showing where you'd been.

If you're worried about how users will react to something innovative you're trying, think about how they understand your purpose. Why did they sign up for you in the first place? Ignore the grand vision in your head, what do they think you do? If what you're doing makes sense for that goal, you'll be surprised at how generous and supportive they can be, even for potentially scary applications. If you're working towards something they don't expect, if you're moving outside of the silo they think you're in, you may be in trouble!

Photo by Philip Chapman-Bell

The Normal Well-Tempered Mind – I never knew the AI community had a favorite philosopher, but I can see why Daniel C Dennett is it. There are so many ideas in this conversation that made me think about how our minds work in a very different light. Even better is his disclaimer: "Everything I just said is very speculative. I'd be thrilled if 20 percent of it was right." That's an attitude I'll try hard to emulate.

Space Station Challenge – Figure out how to eke more power out of the solar panels by carefully changing their positions over an orbit. It's all the constraints that make this coding challenge so much fun.

Understand the favicon – A purist would be appalled, but the hacker within me loves how we're learning to push the limits of what's possible thanks to a deep understanding of platform quirks. Like the space station challenge, a complex but ultimately understandable set of constraints makes fertile ground for artful programming. Check out this beautiful subversion of browser's text rendering engines if you're into that sort of thing.

Pulse Tech Talk 2 – One of the best things about living in San Francisco is the plethora of great tech talks on your doorstep. Check out AirBnB's series too, they have some mind-blowing speakers.

Love and other conspiracies of the X-Files – I have a confession – I've watched all nine seasons, and I'm gearing up to rewatch them soon. They're not all good in a conventional sense, but almost every episode is interesting, and Josh captures some of the roots of why they could be so compelling.