The Planning Fallacy and Software

Dontbelate

The Overcoming Bias blog had a great post on the Planning Fallacy a few days ago. They’ve got some great psych experiments to back the term up, but it boils down to people being inherently bad at figuring out how long a task will take. We always underestimate!

This may sound familiar to anyone who’s worked in software. Guy Kawasaki’s rule of thumb is add six months to the worst case shipping date you’re given, and that sounds right to me. What’s interesting about the post is it not only documents the problem, they also offer a solution.

They describe the normal way people create time estimates as inside planning. This is where you look at the tasks you need to do, create estimates for each of them, and total them to get the final estimate. There’s no time for the tasks you forget, or anything unexpected. What’s interesting is that even when asked to produce a worst-case estimate, people don’t allow enough time.

Their solution is to use outside planning. For this you ignore the unique aspects of the project, and instead try to find a similar completed project, and look at how long that actually took. This usually produces a far more realistic estimate.

This sounds right to me, one of the strengths of an experienced team is that they have a lot of previous projects to compare against. It’s a very strong argument to say ‘well, we’re barely at alpha, and it took us six months to get from here to shipping on project X, so we need to rethink releasing in two months’. If both the team and management were involved in project X, it’s hard to ignore that.

Of course, one of the strengths of a greenhorn team is that they aren’t as cautious! They’re likely to over-commit, but with smart management scaling back the tasks once reality kicks in, they might still produce a better project overall.

Occasionally too, the Captain Kirk/Scotty management/engineering dynamic actually works; "Captain, it will take two weeks to fix!" / "Scotty, get it done in the next thirty minutes." Sometimes that pressure will force a rethink on the engineering side about how to fix something. Maybe there’s a quick hack that will solve 80% of the problem?

Funhouse Photo User Count: 885 total, 72 active daily. I think growth’s actually slowed since my recent changes, so I’ll definitely need to instrument and analyze some usage statistics to try and work out why, and also take a fresh look at the user experience.

Google Hot Keys Download Count: I’ll be occasionally showing my GHK download numbers too, since they’re growing pretty well. I’m up to 4894 total so far on the main Mozilla site. I’m not seeing many IE downloads from my own site, it’s not been approved for CNET downloads yet, and isn’t in any other distribution channels.

Defrag Connector done

Defragprofile

I finished off Defrag Connector. You can now search for friends who are going, any mutual friends you share with those attending, and add a button to your profile.

It’s all unofficial at the moment, but I’m checking with Eric Norlin to make sure he’s ok with me publicizing it a bit more. I wrote it to scratch an itch, but I think it could be handy for other attendees too.

Technically, it wasn’t too hard. The biggest hurdle was doing the mutual friends check, since the Facebook API doesn’t offer a direct method for that. Instead, I’m checking every attendee against every friend of the current user, one attendee at a time. This needs an API call per attendee, so it can take many seconds. Since Facebook will show an error screen if you take too long returning a page, I had to implement an ajax callbacks to deal with a few attendees at a time, and update the page. It’s the same technique I used for to solve Funhouse Photo’s loading time problems. It works well with my few dozen friends, I hope it will scale to some of the monster friend counts some of the attendees have!

I also discovered that you can’t add images directly hosted by facebook to the user’s profile, so instead I had to automatically copy the logo onto my server, and then link to my copy.

One nice bonus of creating this was hearing from Rob Johnson of EventVue, a tech stars company that’s got some great ideas on how to create kick-ass conference networking tools. It’s great to see how many good companies are emerging from the program. Rob told me the secret was how clued-in and involved their mentors are. I’m looking forward to continuing the conversation over a beer at Defrag.

Funhouse Photo User Count: 871 total, 54 active. Still not showing any increase in the growth rate. I think my next step should be to add a lot more instrumentation to the app, so I can tell how people are using it, and try to identify parts of the user experience that aren’t clicking. Incidentally, the total now seems to be updated daily, like the active count. I miss the addictive thrill of seeing that tick up through the day!

Defrag Connector

Defragbuttonpete
I’m getting ready for the Defrag conference in November, and wanted to see which attendees were  friends, or even friends-of-friends. The organizer, Eric Norlin, has created a Facebook event, and a lot of the attendees have signed up.

In Facebook, the only way to interact with that information is by browsing an alphabetical list of the attendees. There’s no way to sort or filter them by their closeness in the social graph, or by shared networks.

This seems like an opportunity to build something better using the Facebook API. My goal is to create a simple application that lets me see attendees who I’m socially related to. That seems both technically straightforward and useful to other people who are either attending, or deciding whether to attend.

I’ll be spending my spare time putting that together, I’m hopeful it won’t take too long to get it functional. So far, I’ve added the application information to Facebook, and I’ve started with Funhouse Photo’s code as a template. The challenging bit is going to be efficiently querying the graph, since it will require a lot more database code than I’ve written before.

Funhouse Photo User Count: 839 total, 98 active. This is a nice increase in the total count, and the active users are up a bit. It’s too early to say if this is related to my changes or just noise, since they’ve been in for less than half a day, but it’s good to see nonetheless.

Funhouse Photo improvements

Scarymonkey
I spent a lot of today working on Funhouse Photo. My first goal was to add notifications to user’s feeds when they updated their status. This was top of my list because it would improve the user experience by spreading the pictures and captions they created more widely. It should also help spread word about the app, so I’ll be keeping an eye on the user figures over the next few days to see if it has an impact.

My old picture choice used a two-step process; first picking the picture, then the caption. I had to cut that down to a single stage so I could add the feed item at the end. To do that, I integrated the caption setting with the picture choice, which left the picking page looking a bit more cluttered, but simplified the flow.

While I was working on that, I discovered a bug that must have been lurking for some time. I was always calling require_login() at the start of my PHP page generation, but for most of what I was doing (adding items to the profile, etc) it made more sense to call require_add(), since that gives the needed privileges. This meant that all the links designed to lure in new users would fail to produce any results, and instead the only way to get the application was to manually add it! This actually made me feel a bit cheerier about my user figures, since they were jumping through more hoops than I thought before they could install the app.

While I was researching that (there’s no documentation on those calls) I also ran across fb:profile-action. This lets you add a menu item below a user’s picture, which is perfect for Funhouse Photo, so while I was working I also put one of those in for folks who’d installed the app. Now when you’re browsing a profile, you will see a ‘Funhouse this photo’ link below the portrait, which I’m hoping will help the engagement level with the app.

As always, I’ll be watching the figures to see what difference this makes.

Funhouse Photo
User Count: 798 total, 59 active. A little worrying, since I think this was the exact total I saw a few hours ago. It seems a bit unlikely that nobody added it in the last few hours, so I’m hoping that it’s a glitch in the stats reporting rather than a bug in the app that’s preventing users from adding. The active total is a lot lower than I’d like, we’ll see if my recent changes help that.

Job Satisfaction

I just returned from a day leading a trail crew with the Santa Monica Mountains Trails Council. Here’s one of the problems we tackled:
Toestubber1

And here’s a close-up:
Toestubber2

The trail had washed away, leaving only a few inches for people to walk on!

Toestubber3

Here’s the solution, after me, Ed and Liz spent half an hour with pick-axes, saws and loppers. A whole new trail section. Ed’s standing where the old trail was, everything to the left is new.

I spend all week hunched in front of a computer, and it can take months or years before I see the final results of a project. That’s why trail work’s so much fun for me; it’s outdoors, physical, and you see instant results.

Funhouse Photo
User Count: 780 total, 70 active. Much the same as before, hopefully I can spend some more time hunched over my laptop tomorrow, adding feed notifications and some new effects.

Developing an IE plugin compared to a Firefox extension

Goatvsgoat

Yvonnick Esnault mailed me to ask how long it takes to develop an extension for Internet Explorer compared to one for Firefox, and where he could learn more about the differences between the two.

I don’t know of many resources that will help, which is why I started my series on porting a Firefox plugin to Internet Explorer. That doesn’t include a summary of the differences though, just an examination of the details, so I’ll try to sum up what I know.

Compiled versus interpreted

IE plugins are Browser Helper Objects, which are effectively special DLLs. These require the use of a compiler, and creating a new plugin or modifying an existing one takes a lot longer than if you’re writing in Javascript for a Firefox extension. It does have one advantage though; you have the option of keeping the source code closed with IE, whereas it’s inherently open-source with Firefox’s Javascript.

Writing an installer

Firefox plugins have a dedicated installer system that you can write some simple scripts to interface with, and use very quickly. You don’t get any help from Internet Explorer, instead you have to write a standard Windows installer executable, which can be pretty time-consuming.

Performance

One other benefit of being compiled rather than interpreted is that processing-heavy operations tend to run a lot faster. I notice this when I’m doing things like string searching within a document.

Firefox development is a well-trodden path

There’s a lot of people creating Firefox extensions, there’s very few creating IE plugins. That has a lot of consequences:

  • There’s much more documentation for FF extensions, both from Mozilla and developers themselves, and it’s easier to get help.
  • There’s fewer obscure bugs in Firefox than IE, because the interface is heavily tested in use.
  • Firefox has addons.mozilla.org to distribute extensions. The IE equivalent, Windows Marketplace is not as well-known or promoted.
  • Since there’s fewer IE extensions than Firefox ones, there’s less competition for users.
  • Unfortunately, many users who want plugins use Firefox rather than IE already because of the lack of IE extensions!

Overall, developing plugins for IE is a lot harder than developing for Firefox, and it took me a couple of months of weekends and evenings to convert mine over. If you do decide to do it, I’d recommend looking at and adding to the BHO documentation wiki.

Funhouse Photo User Count
: 761 total, 68 active. Growth a little slower today, still within the rough range I’ve been seeing for the past couple of weeks.