Photo by Luigi Chiesa

The only way to tell if you're making progress is to measure it. If you're web-based and pre-revenue like me, then traffic and customer reactions are all you have to guide you. Happily Google just released their analytics API, and by combining that with Twitter you can create your own automated daily email report, absolutely free!

If you want to see the results, I'm publicly posting them on the Mailana Stats list:
http://groups.google.com/group/mailana-stats

There's also an RSS version here:
http://groups.google.com/group/mailana-stats/feed/rss_v2_0_msgs.xml

The result is an email sent daily that looks like this:

Subject: twitter.mailana.com daily report: 04-22-09
1673 visits, 1423 visitors
15 twitter messages mentioning mailana

You can grab the source here; it's a set of bash scripts tested on Fedora and OS X. To run it on your own site you'll need to follow these steps:

1- Install Google Analytics on your site
2- Download my scripts onto your Linux machine
3- Get your profile ID. Edit galistprofiles.sh with your Google email address and password, and then run it to see a list of website's stats you have access to and each site's profile ID.
4- Enter your account details. Put your gmail address, password and the profile ID into the gatotalvisits.sh script.
5- Test your API access. Run ./gatotalvisits.sh and by default you should see a single number showing the number of visits. You can look at other stats by passing one of the many metric names as an argument to the script, eg ./gatotalvisits.sh visitors
6- Make sure you can email from the command line. The script needs a way to mail the results, and you may not this set up on you Linux box by default. I ended up following this guide and using SMTP to send from my existing Gmail account.
7- Create a Google Group. You need somewhere to organize and publish your results. My list is public, but you can also create a private version with controlled access. This also gives you an RSS feed, though obscurely you get the link by clicking on the XML button at the bottom of the group's home.
8- Pick Twitter search terms. I'm doing a daily search of Twitter for mentions of 'mailana', but you can edit emailstats.sh to look for your own keywords. Run ./latesttweets.sh <your keyword> to test.
9- Customize the email. Edit emailstats.sh to get the subject and message content in the form you want, and maybe add other stats you care about too.
10- Schedule it daily. I created a link to the script in the daily cron directory by running ln -s /vol/analytics/emailstats.sh /etc/cron.daily/emailstats.sh

You should now have a fully automated daily report email. Let me know how this works for you; I'm avoiding full XML parsing to keep the scripts lightweight so I'll be keeping an eye out for API changes that might break those assumptions.

Photo by Jef Poskanzer

Last week I suddenly noticed the number of imported people on http://twitter.mailana.com/ increase dramatically. This was suspicious since it typically takes a minute or two to import a single person's messages from the Twitter API, so seeing 50,000 added in less than a day rang alarm bells. Simultaneously I got a small flood of emails from users whose profiles were showing up completely blank. Since this is usually the result of a failed import I checked the server logs and there were indeed lots of errors.

Unfortunately I was in the middle of moving house, so I had no time to investigate and fix the problem. Instead I took down the names of everyone who contacted me in my bug database, sent them notes so they'd know I was on the case, and then completely turned off all imports. This meant at least no further profiles would be corrupted before I could solve the problem.

Yesterday we'd finally completed the drive and got the internet running at our new place, so I could sit down and figure out what was going wrong. The immediate cause was this change to the Twitter API on April 9th. Previously I'd been able to use POST for all my calls, but now some would only work with GET. This limitation was always in the documentation but never enforced, so I hadn't spotted it.

That wasn't the true problem though – that sort of changes happens all the time and it shouldn't cause corrupted data and empty profiles. In the spirit of Eric's Five Whys, here's a root-cause analysis:

1. Why did people's profiles show up blank? The import failed and output bogus data when Twitter's API changed.
2. Why did the import fail and output bogus data? The errors weren't detected and handled correctly.
3. Why weren't the errors handled? The import code wasn't tested thoroughly enough.
4. Why wasn't it tested? There was no easy way to run a test.
5. Why was there no easy test? I'd never expected the Twitter import to be so heavily used, it was quickly written code reused from another project.

With that in mind, I worked backwards down the list today, trying to address each layer of the problem in turn. Going in that direction is important because you want to leave the immediate cause until last, so you can verify that the deeper fixes actually do catch that problem.

5. This is a priority issue. I've been trying to juggle my work on email and Twitter simultaneously. That's given me too many top priorities, which really means I have no priorities. To fix that, I'm formally pausing my Exchange work for the next few months. Twitter has become a great platform to showcase my ideas. I still believe passionately that email is a killer application for this, but Twitter is a fantastic way to sell people on what I'm building, once I have people convinced it will be a lot easier to persuade them to invest time and trust installing my email version. This decision will let me give the Twitter code the resources it needs to shine.

4. I built a new unit test into the Twitter import script.

3. That test is now part of my routine whenever that code is changed.

2. I implemented entirely new error catching code. It now correctly halts the script whenever the API returns a fatal error, so no bad data is ever stored in the database. As a bonus, I also now catch temporary errors caused by server overload, etc, and wait 10 seconds and retry a fixed number of times. It's surprising watching the logs how many 502 errors I see!

1. Finally, I switched the API call from POST to GET, and got the import process rolling again.

That wasn't the end of it though. I still had a database with several thousand corrupted profiles. I'd implemented a manual method to force a reimport for users I knew about, but I had to switch to something automatic to handle that number of problems. Another issue was there was no easy way to identify all the affected users thanks to the way I'm storing the data.

I settled on detecting when a blank profile was loaded, displaying an error message then and forcing a full reimport. This is far from ideal, but with the reimport bumped to the top of the queue, should only take a few minutes. If your profile was previously showing up blank, please give it another try, hopefully this will fix that problem for you.

Thanks to everyone who helped me with bug reports on this one, and sorry for those caught with empty graphs for the last week. As always, please let me know about any other issues you're hitting.

We just finished the two day drive from Los Angeles to Boulder, with 3 cats and a dog in my hatchback. It was an exhausting but beautiful journey, especially the Virgin River gorge and pretty much all of Utah. It was a shame it was dark by the time we drove from Grand Junction to Vail, we could tell it must be a wild ride by the way they'd had to build the freeway! Luckily we have one more trip to make that we'll try to time better.

The day after we arrived, the snow started to fall, leaving Boulder looking Christmas-card perfect. One of our big worries has been how our dog Thor will cope with the cold, he's not quite a Beverley Hills Chihuahua but he's definitely grown up used to the southern California weather.

At first he was a bit freaked out by the white stuff that kept landing on his nose, but once he discovered the deer, squirrels and foxes that kept crossing our path while we walked, he had a wonderful time.

This is a fantastic start to our Colorado adventure, thanks to everyone who helped arrange this unseasonal snow. I'm also thankful we weren't trying to make it over the Vail Pass on I70 yesterday!

Photo by Heather

I recently got an email from a graduate student asking for help. He wasn't really a programmer but wanted to get someone's followers and friends in .csv form to feed into his analysis tools. One of the joys of the XML REST api that Twitter uses is that it's all human-readable, so you can get a long way without coding. Here's a quick guide.

The best place to start is this web page:
http://apiwiki.twitter.com/REST+API+Documentation

That
lists all the information you can get from Twitter about other users. All of the API calls are actually web addresses, so you can work
with them without using any code. For example, if you type this into
the address bar of Firefox or Safari (not tested on IE) you'll see a
formatted list of the IDs of all my friends:
http://twitter.com/friends/ids.xml?screen_name=petewarden

The same idea works for followers:
http://twitter.com/followers/ids.xml?screen_name=petewarden

The output is actually XML, but you don't need to understand the details to pull out simple information. It's not quite CSV, but hopefully with some simple
text replacement hacking you might be able to convert it to the form
you want. For example you could copy and paste from Firefox and use your favorite text editor to remove "<id>", "</id>", "<ids>" and "</ids>". You'd then have a text file with just an id number on each line.

The next issue would be mapping those ids to names. To do
that you'll need another call, replacing 4411041 with the number you
want:
http://twitter.com/users/show/4411041.xml

In Firefox you'll see another page of XML. To get the information you need, search for the line with "<screen_name>".

You'll
also probably want to do this from the command line to automate the
process which will mean some coding. I recommend using a unix-y system
like OS X or Linux, since they give you access to curl to fetch the web
page rather than manually copying it from the web browser, and the text
processing tools (eg Perl or your favorite script language). If you're
stuck on Windows, wget and .bat files may be an alternative, though it
won't be pretty.

Another wrinkle is that some of the calls require logging in. One way to do that is include your username and password in the URL, something like this:
http://username:password@twitter.com/&#8230;

Photo by Difusa

Most people imagine that introvert is a synonym for shy, but it's more accurate to say that they're someone who is recharged by time spent alone, and drained by company. That definition fits me – it's not that I lack social skills or dislike time spent with others, I just have a limited stamina for social gatherings and thrive in one-to-one conversation and quiet time.

When I saw this old Atlantic article on caring for your introvert it rang very true. I don't agree with the complaints about oppression – in the computer industry introverts are well catered for – but if people understood more about introversion life would be a little easier. My friends have got used to the fact I like nothing better than just quietly hanging out with them, but that's confusing for a lot of people I meet. I'm not shy, and don't mean to come across as arrogant or stand-offish, I just lean towards a high signal-to-noise ratio in conversation, which means a high thinking-to-talking ratio too!

Photo by Bill McIntyre

A lot of my favorite technologies are allegedly dead. You don't get more unfashionable than Usenet, but I was just able to get an answer to my story ID request in under 10 minutes from rec.arts.sf.written! Show me a web tool that can match that. Email's another technology that's been written off, but it's still the central electronic communication channel for most people.

I'm not a mindless luddite, I love shiny new toys as much as the next geek, but I try to learn from how people actually use computers, rather than how I'd like them to. I remember an Enterprise 2.0 technologist describing how he used a wiki to create all his documents, and was frustrated that the rest of his company wouldn't do the same. He didn't get that while wikis are a great innovation, as word processors they suck.

Users generally push back on changes for a reason. If you're getting strong resistance that seems senseless, that just means you don't understand their requirements well enough. Go back and stare deeply at how they use the old solution. You'll usually see why they keep dragging that corpse around.

One of the toughest parts of our relocation to Boulder is moving away from Santa Cruz Island. It's a 100 square miles of old California, just an hours boat ride off the LA coast. Our last trip out was to help the rangers with some trail maintenance for a few days, and one of the crew was an old friend Dave Edwards. I learnt that Dave had been coming out to the island for 25 years, long before it was a National Park while it was still a private ranch. He had a CD full of photos dating back to 1987, and we had a fascinating evening flicking through them. Part of my fascination with the place is watching the transformation as nature recovers from the ranching of the last century. Every trip we see more lush native vegetation and animals now the pigs and sheep have been removed. You can see the changes even more clearly in Dave's photos.

Also staying in the ranger's huts was another David, a research biologist working to re-populate the small Scorpion Rock with native plants. He explained how a tourist snap from the 1940's had been the key to figuring out what the original native mix was, before the invasive species wiped them out. That got me thinking, Dave's photos might be useful for something we'd never imagined, if only they were available to the right people. The best way I know of getting them out there is throwing them onto the internet, so I got his permission to upload them to Flickr. Hopefully future students of Santa Cruz Island will find some valuable insights, while the rest of us can just enjoy flicking through them.

http://www.flickr.com/photos/petewarden/sets/72157616449754152/

Today I picked up the keys to a little house in Boulder and started moving in. I was initially very confused by all the white stuff falling from the sky (we don't get that in Los Angeles) but I'm slowly adjusting. The day has mostly been a blur of unpacking and floor-mopping, but the neighborhood deer paid me a charming welcome visit, ambling slowly through our front yard. I was very surprised, we're well into the town (near Norris and 21st), but according to the locals this isn't uncommon.

A trip to the local Safeway produced another unexpected delight:

Unlike any mainstream California supermarket they have an entire section devoted to British food! Maltesers have pride of place (we've proven by extensive experimentation in our D&D group that Whoppers are far inferior), the gap in the Dark Chocolate Digestives (terrible name, divine cookies) is from me snatching a packet, and they even have cans of the appalling but amusingly named Spotted Dick dessert. There's Heinz Baked Beans, the only brand for authentic Beans on Toast (the name is the recipe!), pickled onions (horrid and stinky), and most dangerous of all, Liquorice Allsorts. I see some serious dental bills in my future with this sort of temptation…

I'm working on a short description of where my business is at, and I'm publishing what I have so far, in the hope I'll get ideas on improving it. The audience is potential investors and other partners, and the goal is to just to start a dialog, so I'm keeping it short and snappy. Let me know in the comments what you'd think if this landed in your inbox.

Mailana Inc
"You guys should talk"

Mailana produces actionable information from your
electronic conversations. It analyzes email and IMs to answer questions
like "Which of my friends know this person?", "Who in my circle knows
about this subject?" and "How can I connect with somebody at this
organization?"

The public Twitter demo has won praise from people like Tim O'Reilly, Brad Feld, Brian Solis of TechCrunch and Marshall Kirkpatrick of ReadWriteWeb.

The system also runs against Microsoft Exchange,
downloading and analyzing the email messages for a team, department or
an entire organization. The information is then used to create
suggested profiles for each employee, forming the basis for a much more
detailed internal version of LinkedIn. Exposed through Outlook and
Internet Explorer add-ins, this stops companies reinventing wheels and
unlocks wasted potential resources within the business.

Mailana Inc was founded in July 2008 by Pete Warden,
previously a Senior Engineer at Apple. Its goal is to improve the world
by getting the right people talking to each other. With a fully working
code-base that's been deployed in real companies, it's currently
looking for partners to help as it moves from a science project into a
commercial proposition.

contact: pete@mailana.com

blog: http://petewarden.typepad.com/