Photo by verhoogen.be

I love Joel Spolsky's latest column on the secrets to startup success, it ties together so many of the things I've been wrestling with as I get Mailana off the ground.

The key metaphor he uses is that any software development is like tuning a fiddly shortwave radio. You spend a frustrating amount of time fiddling with the dial listening to static (which is actually pretty mind-blowing once you know it's the echo of thunderstorms on other continents and exploding galaxies). Finally you hear a faint voice, and by carefully tweaking it gets louder. Eventually you get it just right and the station comes through loud and clear.

For a startup the signal you're searching for is customers saying "I want what you're offering". As I know from personal experience working on a long list of projects that I thought were fantastic but never resonated with a large audience, getting that response is hard. That's why getting initial users is so crucial, people are such complex beasts you can't tell what will work until you try it on some volunteers. It's also why listening intently to their reactions is vital. You won't get it right first time but you'll get hints on what you're doing wrong and the right direction to go in.

The other lesson Joel lays out is one that has stuck in my head since reading Paul Graham's essay: Your startup won't die as long as you keep coding. Every line of code represents a chance to tweak that dial, and as long as you're searching you have a chance of hitting that sweet spot. I'm hoping that the combination of frequent changes and listening to feedback will help me constantly improve Mailana for Twitter.

Thanks to everyone who's been giving me brickbats and bouquets on twitter.mailana.com, I hope this explains a little of why I'm so eager to hear back from you all!

Photo by Jeff Bauche

mailana.twitter.com has imported all the messages from 250,000 of the most active Twitter users, which gives me the chance to answer some interesting questions. Who's getting the most out of the service? From the coverage you'd think tech celebrities but the top 50 relationships include none of the usual suspects. Instead it's everyday people talking about their lives in public. UKMelia and Ernmander have exchanged over 2000 messages, and Krowed from Tokyo has two of his friendships in the top 5, but only 68 followers.

Here's the top 10:

1.		ukmelia				ernmander
2.		stefanomuti				milaluna
3.		dragon_harrower				nerdynerak
4.		m_non				krowed
5.		hitmix				krowed
6.		sabell				nerdynerak
7.		shaunduke				mulluane
8.		wherever				devilbesideyou
9.		carnager				makochan200
10.		darkfairymomma				corruptor

How is the chart ranked? I count the number of conversations between every pair of people on Twitter, looking at how many @ messages from one person include the other's name. To score the strength of a relationship I'm using the minimum of the number of messages sent in either direction. So, according to my data these are the people who send the most tweets to each other out of the millions on Twitter, quite a distinction!

The best part of blogging is the comments. My posts are usually about problems I'm wrestling with, so the ideas and reactions are incredibly useful. Unfortunately I went through a couple of rocky weeks with Intense Debate and lost some comments. Thanks to ID's Michael Koenig I'm back in business, but I wanted to quote one from Gist's Steve Newman that I thought deserved a post of its own. He commented on my Get Customers:

I
completely agree with this point. Exercise the app, label it whatever
you want (alpha, beta) but get it in the hands of users and setup the
necessary feedback framework to collect input and take action. On a
similar note, exercise your business model as soon as reasonable too.
Too often, startup companies keep the first day of revenue collection
on the horizon. Until you really start collecting dollars, you wont
know whether that business plan with the hockey stick growth really
does have a basis in reality. Being successful as a startup has
everything to do with building a product that people USE. Building a
business has everything to do with being nimble, adjusting and striving
to hit your goals in the real world, not in excel.

I couldn't agree more, though it can be a tough discipline to stick to! I always think of the hierarchy as feature, product, company. The key test of whether you're building a living company or just some cool technology (that only has a future if it's rolled into someone else's business model) is if you can persuade people to hand over cold hard cash for it.

Photo by Riot Jane

Robert Shippey asked me a simple but intriguing question: What does a 'good' social network on Twitter look like? What makes it so tough is that it depends on what your goals are. Here's a couple of different answers:

You need a network that's very tightly interwoven with thick links, where most of your contacts talk to lots of other people you know. The advantage of this structure is that it promotes trust, since if you betray anyone in the group they can retaliate by trashing your reputation with all your mutual friends. The deeper level of trust makes it a very safe environment to build friendships and collaborate.

That high cost for any transgressions has a flip side though. It's like a small town, the same mechanism also punishes non-conformity so it's hard to innovate or do anything controversial.

The most influential people have networks that connect lots of different groups of people. That means they tend to look like a wheel; the influencer at the center with links radiating out to people who are otherwise unrelated. The advantage of diversity is that you can act as the bridge between unconnected communities, so when a good idea appears in one area it can be transmitted to others where it could be useful.

Ronald Burt uses a closure metric to measure how tight-knit your network is. In simple terms, it's the chance that for any random person in your network, they'll know a second random person from your contacts. If everybody in your circle talks to everyone else, then that probability will be 100%, but if you're only connecting people who otherwise don't know each other it will be zero. I might add that to the profiles to give people an idea of where they fall on that spectrum.

Photo by DerrickT

One of the most annoying bugs with twitter.mailana.com has been the missing images that show up in the graphs. I'm only able to scan Twitter every few days, and when people change their profile pictures the old links no longer work.

The ephemeral nature of the portrait links is a known shortcoming of the API, and I was getting a headache thinking about the possible ways to work around it. I'd either have to scan more often and tax both my and Twitter's servers, or creating a dedicated service that would handle the housekeeping of making sure the URLs stayed valid.

Then I discovered Shannon Whitley's SPIURL project, which offered a free static URL for any user's portrait image. Just link to http://purl.org/net/spiurl/<username&gt; (eg http://purl.org/net/spiurl/petewarden ) and you'll get the image data back for that user. I switched my graph viewer over to using those URLs, and now you should never see a missing image again. Thanks so much Shannon for implementing this!

Photo by Brit

I realized yesterday that "You guys should talk" is the perfect tag line for Mailana. For almost anything you're trying to do, there's someone else out there who can help you and benefit themselves from getting involved. I've seen this so many times my life, win-win situations if you can just make the right connection between two people.

I'm a naturally pretty introverted person, if you use Brad's definition of an introvert as someone who recharges their batteries by spending time alone. While I like to think I'm still pretty sociable, I know how tough it can be to build up enough background knowledge on people to be able to suggest fruitful connections.

Mailana is about helping people connect by making it easy to share relevant information on yourself so you can find and be found by others. The best source for that description of what and who you know is your email inbox.

There's massive value to be unlocked in companies, and in our everyday lives from just being able to connect people. You can already see glimmerings of this in social networks like Facebook and LinkedIn which offer computer-aided ways to strengthen and navigate your relationships. Using the implicit data from our existing communications (like I'm experimenting with at twitter.mailana.com) we'll be able to build far more effective tools to link people together.

Photo by Junjan

One of the hardest things to do when you join Twitter is figuring out who to follow. Once you've added your immediate friends a good next step is people who live locally. There's a variety of ways to do this – first give http://twitter.mailana.com/find.php a try. Enter the name of your city or town, and it will bring up a list of everybody who's recently Twittered from there. It's a stepping stone to graphing the social networks for every location, so stay tuned.

Another good way to find local people is to use Twitter's built-in search. Enter your location in the 'Near this place' field and it will give you recent messages from people nearby.

To get a list of the most active and followed users in your neighborhood, go to http://twitter.grader.com/location and search for your home town. You'll see a ranked list, it's good to pick up on any local net celebrities you might not know about.

Photo by Nebarnix

Twitter's Search API is fantastic, but unlike the UI version they don't support using the name of a place to limit your results, only latitude and longitude. I couldn't ask my users to type in coordinates for searches, so I needed some way to translate the names into the form Twitter wants.

The process of converting from human names into coordinates is known as geocoding, and luckily there's some great free web APIs that will do the job for you. I chose Yahoo's GeoPlanet, they allow up 50,000 requests a month, and Tyler Hall has a great PHP wrapper ready to use.

I couldn't find a good example of actually using GeoPlanet in PHP, so I decided to kill two birds with one stone and publish my code as sample code for geocoding, and offer an off-the-shelf solution for anyone else who needs 'near' for Twitter searching. You can download geosearch.php
, and it's included below.

<?php

// Example demonstrating how to use Yahoo's Geoplanet API (via Tyler Hall's PHP wrapper)

// to emulate the 'near' parameter that's available in the Twitter search UI, but not

// through the API.

//

// To install it:

// 1) Put this file and class.geoplanet.php on your server

// 2) Get a free application ID from Yahoo – http://developer.yahoo.com/geo/

// 3) Copy the ID where this file has <Put your own app id in here>

// 4) Remove the die() statement just above it

//

// To use it:

// Call the URL of the file as you would http://search.twitter.com/search.atom

// All the normal supported arguments are passed through to Twitter's API

// If 'near=' is specified, then the place name string is passed to Geoplanet

// The latitude and longitude of the result are passed as 'geocode=' to Twitter

// If you also specify 'radius=' then that is passed, otherwise '25km' is used

//

// Gotchas:

// – It uses a redirect header, so for command-line curl you'll need -L to follow

// – There's zero error or exception handling

//

// Example:

// http://yourserver.com/geosearch.php?q=biking&near=boulder&radius=15km

require_once("class.geoplanet.php");

// If the named parameter was present on the input URL, append it to $url

function copy_url_input($name, $url)

{

if (isset($_GET[$name]))

$url .= "$name=".urlencode($_GET[$name])."&";

return $url;

}

$searchurl = "http://search.twitter.com/search.atom?&quot;;

// If the near parameter is present, then call Geoplanet to turn it into lat,long

if (isset($_GET['near']))

{

$placename = $_GET['near'];

die('You need to put your own app id here and then remove this statement');

$geoplanet = new GeoPlanet('<Put your own app id in here>');

$placelist = $geoplanet->getPlaces($placename);

$topplace = $placelist[0];

$centroid = $topplace['centroid'];

$lat = $centroid['lat'];

$lng = $centroid['lng'];

// The radius parameter is optional. If it's not present then default to 25km

if (isset($_GET['radius']))

$radius = $_GET['radius'];

else

$radius = '25km';

$searchurl .= "geocode=".urlencode("$lat,$lng,$radius")."&";

}

else

{

$searchurl = copy_url_input('geocode', $searchurl);

}

$searchurl = copy_url_input('q', $searchurl);

$searchurl = copy_url_input('lang', $searchurl);

$searchurl = copy_url_input('rpp', $searchurl);

$searchurl = copy_url_input('page', $searchurl);

$searchurl = copy_url_input('sinceid', $searchurl);

$searchurl = copy_url_input('show_user', $searchurl);

// Request a redirect to the URL with the lat,long added

header("Location: ".$searchurl);

?>

Photo by Suchitra Prints

http://twitter.mailana.com/ has been in the wild for a week now, and the best part has been hearing from users. I'm only human, so of course I love it when people tell me how well it's working; but as a developer the best comments are the complaints. Here's an example of why they're so useful.

I have a search filter set up for Mailana so any Twitter messages mentioning it show up in my RSS reader. On Saturday evening I spotted this from @mayaREguru

That Mailana thing is way wrong wrong wrong and weird. Makes no sense at all

The first thing that caught my attention was that she wasn't just hitting some minor glitches, it sounded like it was completely broken for her. That made my ears perk up, because if others are hitting similar problems it will be ruining a lot of people's experiences. Most of those people won't complain, so a single comment likely represents tens or hundreds of unimpressed potential users.

I got in touch with Maya and she gave me some more details. The whole BFF list was showing up incorrectly, with a lot of people she frequently talked to missing, and other people she'd only exchanged a few messages with listed instead. It's actually a lot easier to debug a massive problem like that than something more subtle, so I set to work figuring out what was going wrong.

Mailana works by mapping conversations. To know what conversations someone's had in Twitter, I need to import all the status updates for both sides of the conversation. That means for any user, there might be hundreds of other people they've sent @ messages to, and I need to download all their status updates to discover if those other people replied. When a new graph needs to be generated for a person, I first grab all their own messages, and then import all the messages of everyone they've sent messages to.

When I looked at Maya's information in the database, I could see there were a lot of people she'd sent messages to who hadn't been imported, so as far as the system was concerned she'd not had conversations with them, since it never saw any replies. I ignore those one-sided exchanges, since I'm looking for real relationships where there's two-way communication, so all those unimported people weren't showing up in the list or the graph.

Now I knew the immediate cause of the problem, but I had to work out why those people hadn't been imported. I won't go into the gory details, but after some debugging it turned out a crucial bit of code was looking at the wrong piece of information when figuring out the list of people to grab messages for.

It turned out my initial guess was correct, the same problem would have affected a lot of people's profiles. So by fixing Maya's graph I solved it for them too.

The issue never showed up in my own testing and I'd still be silently losing potential users still if I hadn't heard from her. That's why bug reports are like gold dust.

When I was working in the game industry writing tools for artists, I'd literally be sitting back-to-back with my users. Being that close to my customers made me feel their pain (and hear their cursing). That guidance helped me produce the programs I'm proudest of. Knowing they could turn around and slap me on the back of the head when it crashed concentrated my mind wonderfully!

Thanks to @mayaREguru, @robertshippey, @aleferreira and everyone else who's told me about things going wrong. Keep the complaints coming, it's really helping me improve Mailana!

Photo by OneEighteen

I wanted to add a tag cloud of all the terms people had used in their exchanges, to give an idea of what any two people were talking about. Normally I'd just query my Mailana system directly for the messages' content, but since my server is pretty busy importing new people and generating graphs, I thought I'd try Twitter's search API on the client side, using JSON to load the results.

I'm now using this on http://twitter.mailana.com/ to generate the clouds you see on the bottom left when you look at somebody's profile, but to make it more understandable I've created a standalone example at http://web.mailana.com/labs/twitcloud/

It works by constructing a new script element within the document pulling from the JSON search location. You can supply a callback function name, and that gets called back with the results of the search. View the source to understand what's going on, you're free to reuse any of this code in your own projects with no restrictions.

The biggest drawback is that the search API has been somewhat unreliable for me today, sometimes returning server 500 errors repeatedly, which results in a garbled callback. Despite that I'm still blown away by how much Twitter exposes through their API, it's a developer's dream.