Photo by Clio20

I've just spent several days tracking down a serious but hard to reproduce bug, so I wanted to leave a trail of Googleable breadcrumbs for anyone else who's hitting similar symptoms.

As some background, Jetpac's iPad app uses a UIWebView to host a complex single-page web application. There are a lot of independent scripts that we normally minify down into a handful of compressed files in production. Over the last few weeks, a significant percentage of our new users have had the app hang on them the first time they loaded it. We couldn't reproduce their problems in-house, which made debugging what was going wrong tough.

From logging, it seemed like our app setup Javascript code was failing, so the interface never appeared. The strange thing was that it was rarely the same error, and often the error locations and line numbers wouldn't match with the known file contents, even after we switched to non-minified files. Eventually we narrowed it down to the text content of some of our <script> tags being pulled from a different <script> tag elsewhere in the file, seemingly at random!

That's going to be hard to swallow, so here's the evidence to back up what we were seeing:

We had client-side logging statements within each script's content, describing what code was being executed at what time, combined with <script> onload handlers that logged what src had just been processed. Normal operation would look like this:

…

Executing module storage.js

Loaded script with src 'https://www.jetpac.com/js/modules/storage.js&#039;

Executing module profile.js

Loaded script with src 'https://www.jetpac.com/js/modules/profile.js&#039;

Executing module nudges.js

Loaded script with src 'https://www.jetpac.com/js/modules/nudges.js&#039;

…

In the error case, we'd see something like this:

…

Executing module storage.js

Loaded script with src 'https://www.jetpac.com/js/modules/storage.js&#039;

Executing module profile.js

Loaded script with src 'https://www.jetpac.com/js/modules/profile.js&#039;

Executing module storage.js

Loaded script with src 'https://www.jetpac.com/js/modules/nudges.js&#039;

…

Notice that the third script thinks it's loading nudges.js, but the content comes from storage.js!

Ok, so maybe the Jetpac server is sending the wrong content? We were able to confirm through the access log the file with the bogus content (nudges.js in the example above) was never requested from the server. We saw the same pattern every time we managed to reproduce this, and could never reproduce it with the same code in a browser.

As a clincher, we were able to confirm that the content of the bogus files was incorrect using the iOS 6 web inspector.

The downside is that we can't trigger the problem often enough to create reliable reproduction steps or a test app, so we can't chase down the underlying cause much further. It has prompted us to change our cache control headers since it seems like something going wrong with the iOS caching, and the logging has also given us a fairly reliable method of spotting when this error has happened after the fact. Since it is so intermittent, we're triggering a page reload if we do know we've lost our marbles. This generally fixes the problem, since it it does seem so timing dependent, though the hackiness of the workaround doesn't leave me with a happy feeling!

If you think you're hitting the same issue, my bet is you aren't! It's pretty rare even for us, but if you want to confirm try adding logging like this in your script tags, and log inside each js file to keep track of which you think is loading:

<script src="foo.js" onload="console.log('loaded foo.js');"/>

In foo.js

console.log('executing foo.js');

…

Comparing the stream of log statement will tell you if things are going wrong. You'd expect every 'executing foo.js' to be followed by a 'loaded foo.js' in the logs, unless you're using defer or async attributes.

Photo by DJ Badly

How long you spent on it.

How hard it was to implement.

How clean your architecture is.

How extensible it is.

How well it runs on your machine.

How great it will be once all their friends are on it.

How amazing the next version will be.

Whose fault the problems are.

What you think they should be interested in.

What you expected.

What you were promised.

How important this is to you.

I have to keep relearning these lessons. Finding an experience that people love is far more precious and rare than most of us realize.

Team Jetpac from JetpacApp on Vimeo.

Here's a timelapse of thirty minutes in the life of the Jetpac team. You may notice I spend most of my time talking, which is an accurate reflection of my life as a CTO!

Photo by Sanchtv

How Zappo's user agreement failed in court – A lot of people have an almost-mystical belief in the power of terms of service and license agreements. Courts generally look for some minimal evidence that people are reading and understanding what they're agreeing to before they'll agree to enforce those terms. They also don't look kindly on agreements that allow one party to change the terms at any time, without notice or assent. The law may be an ass, but there are humans at the controls and you can't get too far from common sense before they'll intervene.

Free Foundation Trilogy Audio Books – Every now and again I have to stop and be amazed at the cornucopia of art that the internet has made instantly available to me. I remember struggling to piece together book series from libraries and stores, in a way that the web has made obsolete. I was reminded of that when I ran across this eight hour epic production by one of my favorite childhood authors, though I'm slightly afraid to listen in case Asimov hasn't aged well.

The Evil Bit RFC – The perfect antidote to endless angels-on-a-pinhead debates around web specifications (*cough* webfinger *cough*). It's way more fun to sketch the rules of a perfect world than to wrestle with the tradeoffs of an ugly one.

Unwise Microwave Experiments – Who knew you could use your microwave to create lava? I'll be trying these shortly before I move out of my current apartment.

Mind your nanoseconds – It's pretty thrilling to know that we're now doing things so fast in computing that the speed of light is becoming a significant barrier, and Admiral Grace Hopper does an amazing job turning very abstract ideas into something very concrete.

Photo by Lovestruck

There are lots of examples of big companies A/B testing their way to greater success, but it's hard to figure out how to get started with event data as an early-stage company. For one thing, you probably don't have very much data to go on! I'm going to talk about a few of the things I've learned at Jetpac as we've built something from nothing.

Sample size matters

We're still focused on getting a user experience people love, and a value proposition they understand, before we attack distribution. That means we have hundreds of users a day, not thousands or tens of thousands. My initial approach was to try a feature experiment, look at the results after a couple of days and then decide which was more successful. I rapidly discovered that this wasn't working, as a week later the statistics on a feature's usage might be much worse. Looking at measures like the number of slides viewed, it became clear how big the natural day-to-day variation was, often plus or minus 50%!

It's easy to forget basic statistics like I did, but the sample size is really, really important. There are robust methods to figure out the confidence of your results, but for my applications I've found anything less than a hundred is useless, a few hundred becomes indicative, and over a thousand is reliable.

This doesn't often come up because optimizing based on event tracking doesn't usually happen until you have large numbers of users, so it's easy to gather big samples quickly. For us, it's meant that we can only run a limited set of experiments, so we have to be very careful about how we choose the most important hypotheses. It also means that we tend to try out new features on 100% of our users, and compare against historical data, since that's the fastest way to gather samples and time is our scarcest resource!

What people do doesn't always reveal what they want to do

One of the most productive outcomes of having a rich set of data about how our users behave is that we've learned to argue about data, rather than opinions. When somebody has a product idea, we can dig into the existing data and see what evidence there is to support it. A lot of the time this is effective, but the approach has some subtle flaws. We still try to prototype some of the ideas on users, even when existing behavior seems to rule against them, and a few of them work despite the data. Sometimes we've figured out that what the old data is really showing that people didn't use a similar feature because its placement in the interface was poor, or they didn't understand what the text meant, or the icon was unattractive.

People are funny beasts, and looking at how they're using an application and saying, "they want to do X and Y, but Z isn't popular" is helpful, but not sufficient. Sometimes a small variation on "Z" will make a big difference, they may actually want to do it but be discouraged by the way it's been presented. At a big company there's probably a lot of institutional knowledge about what's worked in the past, but you won't have that as a small startup.

There's no substitute for talking to users

So if data's not the whole answer, what can you do? I've been lucky enough to work with a very talented user experience designer and one of his super-powers is sitting down and watching new users try out the app, and talking to them about what they're thinking. A classic example of how this can work is mis-swiping. Our whole app is based around touch-swiping through hundreds of travel photos your friends have shared with you, and he noticed that a lot of people seemed to be having trouble moving to the next picture. He mentioned this at a standup, and I was intrigued. I queried our activity data, and discovered that over 8% of swipes that were started in the app weren't seen as vigorous enough to result in a slide advance! This matched what he was seeing, so we redesigned the swipe gesture to be more sensitive. After we made the changes, we videoed a couple more new users and saw that they had a lot more success. A few days later I had a big enough sample size to be confident that the percentage of mis-swipes had dropped to just 2%!

Guerilla user-testing

User testing sounds awesome, but don't you need a lab to do it properly? I'm sure that would help, but we've managed to get a long way with some simple approaches that anyone can adopt. Bryce's killer technique is tilting a laptop screen down so it's looking at an ipad on the desk, and then having the user play with the app while he records the video of their hands and the audio. I know bringing folks into the office can be very time-consuming though, so we've also ended up getting a lot of value out of in-app surveys from Qualaroo, and simple user tests from UserTesting.com. The videos you get out of the latter can be incredibly useful, and it's a godsend being able to put in a request and within an hour have several fresh users work through any part of your application you want tested.

Knowing in real-time how people are using Jetpac has been amazingly helpful, it's been an incredible learning experience. Even if you're a small fry why not dive into your event data yourself?

Photo by Yesica

I'm embarrassed to admit that I've lost the compiled versions of my open-source visual-effects plugins! In a recent overhaul of petewarden.com I managed to overwrite the binary versions, and while the source code is still available, I no longer have the After Effects and other SDKs and compilers I'd need to rebuild them. They're over twelve years old now and don't work with recent AE versions, but I still get regularly get requests and questions about them, so could anybody who has downloaded binary copies of either the FreeFrame, VisualJockey or AfterEffects plugins drop me an email to pete@petewarden.com? Any help much appreciated!

Hello World in Common Crawl – I've been very excited to see the wider developer community starting to realize how many awesome projects Common Crawl makes possible. Here's a simple guide to getting started with its four billion web pages.

Remember the memristor – A great explanation of an intriguing new computing component, along with a clear-eyed look at why so many of these early-stage experiments never make it into commercial production.

ZIP code data hacking – It's amazing how many of the fundamental facts about our world are kept in closed-license silos. Anybody who wants to link US data sets pretty quickly runs into the translation issues between different area designations, so this project to convert between the common ZIP codes and the governmental FIPS designations is a fantastic idea.

PlacePugs – The ultimate image placeholders, as featured above.

Sorting one million eight-digit numbers in 1MB of RAM – So much of coding is about spotting exploitable patterns in your input data and requirements.

Photo by Joey Rozier

One of the most fun parts of working on Jetpac has been following Cathrine’s stellar posts on the company blog. She’s had some amazing finds, including surprises like the hammock tent! We’ve had hundreds of thousands of visitors and a lot of links, but until recently I didn’t realize that having a separate subdomain at blog.jetpac.com meant that we didn’t get the full search ranking benefit. We’re hosted on Tumblr, so I assumed moving things over to a subdirectory would be fairly straightforward, but it turned out to be surprisingly tough. In the end I not only had to manually proxy all /blog content through our frontend servers, I even had to rewrite the page content on the fly to remove hard-coded links! I found a couple of examples in PHP, but nothing that worked with our Ruby and Sinatra setup, so here’s what we ended up with:

https://gist.github.com/petewarden/3950261

In brief, this code intercepts all URLs that start with /blog on a site, and pulls the content from the equivalent yourblog.tumblr.com. If the page is HTML, it also does some quick-and-dirty regex alterations to change any root URLs in links and resource references to point to the new /blog versions. Not my most elegant bit of code, but it seems to be doing the job.

Photo by Rennett Stowe

Save vs Death – As a gamer, and especially as a role-playing gamer, I've spent a lot of time learning to viscerally understand probabilities at a gut level. I'd never made the connection, but that perspective has informed so much of what I've done in my life, from optimizing products to making life choices. It's sobering to read Jim's meditation on how that approach affects your world view when you're facing mortality head on.

It’s not what you know, but who you know: The role of connections in academic promotions – A natural experiment that demonstrates how much having a personal connection to a job candidate affects decision makers, with some thoughtful analysis of why that sometimes makes sense at an individual level, even when it results in overall unfairness.

Mapping with Geocommons and OpenHeatMap – Out of all the open-source projects I've done, OpenHeatMap has had the longest life. It's a painful mess of PHP code, but it seems to meet a real need, so I try to keep it up and running. The hardest part is that as more and more people use it, the mean time between server failures shrinks, so I end up having frequent down time. I've been working on strategies to reduce the flakiness (the usual failure mode is the postgres server hitting memory problems and dying) but apologies to anyone who's been affected recently.

How location technology can change publishing – My friend Drew Breunig always has interesting things to say about practical applications of location data.

Zombie.js – After the shame of having the Jetpac login system down overnight without noticing, I've been investing heavily in automated testing. So far desktop Selenium has been my favorite approach, but someone pointed me at Zombie as an alternative, and it looks very impressive. I've used PhantomJS before for screenshots, but from what I've seen so far Zombie doesn't require X windows or other heavy dependencies. I'll give a full report once I've had a chance to use it in production!

Photo by Supermuch

One of the hardest things about the recruiting process is having to turn down great potential hires because they're not a good fit with the positions we have. Most of my time at Jetpac these days is spent doing recruiting, so I've ended up with quite a few job-seekers I've been really impressed by, but that I can't hire myself. I've been sending them on piecemeal to friends, but I decided to set up something a bit more organized, so here's a mailing list you can join to see them:

https://groups.google.com/d/forum/youreallyshouldhire

Here's how the group works:

 - You must have a candidate's permission to post their details!

 - Anyone can post candidates, I just ask that you've at least emailed or spoken to them on the phone, to do the initial "Is she a serial killer?" screening.

 - There's no blind resumes, every post should give the full contact details for the candidate. 

 - This is focused on Bay Area tech startups. Feel free to start your own for other areas and industries, with my blessing.

 - This will only add value if it's mostly folks who are actively hiring at their companies, so I'll be moderating the list, hopefully with a very light touch.

If you're a jobseeker and want to be shared with some interesting tech startups, contact me through our jobs page and mention 'youreallyshouldhire'!