Photo by Riot Jane

Robert Shippey asked me a simple but intriguing question: What does a 'good' social network on Twitter look like? What makes it so tough is that it depends on what your goals are. Here's a couple of different answers:

You need a network that's very tightly interwoven with thick links, where most of your contacts talk to lots of other people you know. The advantage of this structure is that it promotes trust, since if you betray anyone in the group they can retaliate by trashing your reputation with all your mutual friends. The deeper level of trust makes it a very safe environment to build friendships and collaborate.

That high cost for any transgressions has a flip side though. It's like a small town, the same mechanism also punishes non-conformity so it's hard to innovate or do anything controversial.

The most influential people have networks that connect lots of different groups of people. That means they tend to look like a wheel; the influencer at the center with links radiating out to people who are otherwise unrelated. The advantage of diversity is that you can act as the bridge between unconnected communities, so when a good idea appears in one area it can be transmitted to others where it could be useful.

Ronald Burt uses a closure metric to measure how tight-knit your network is. In simple terms, it's the chance that for any random person in your network, they'll know a second random person from your contacts. If everybody in your circle talks to everyone else, then that probability will be 100%, but if you're only connecting people who otherwise don't know each other it will be zero. I might add that to the profiles to give people an idea of where they fall on that spectrum.

Photo by DerrickT

One of the most annoying bugs with twitter.mailana.com has been the missing images that show up in the graphs. I'm only able to scan Twitter every few days, and when people change their profile pictures the old links no longer work.

The ephemeral nature of the portrait links is a known shortcoming of the API, and I was getting a headache thinking about the possible ways to work around it. I'd either have to scan more often and tax both my and Twitter's servers, or creating a dedicated service that would handle the housekeeping of making sure the URLs stayed valid.

Then I discovered Shannon Whitley's SPIURL project, which offered a free static URL for any user's portrait image. Just link to http://purl.org/net/spiurl/<username&gt; (eg http://purl.org/net/spiurl/petewarden ) and you'll get the image data back for that user. I switched my graph viewer over to using those URLs, and now you should never see a missing image again. Thanks so much Shannon for implementing this!

Photo by Brit

I realized yesterday that "You guys should talk" is the perfect tag line for Mailana. For almost anything you're trying to do, there's someone else out there who can help you and benefit themselves from getting involved. I've seen this so many times my life, win-win situations if you can just make the right connection between two people.

I'm a naturally pretty introverted person, if you use Brad's definition of an introvert as someone who recharges their batteries by spending time alone. While I like to think I'm still pretty sociable, I know how tough it can be to build up enough background knowledge on people to be able to suggest fruitful connections.

Mailana is about helping people connect by making it easy to share relevant information on yourself so you can find and be found by others. The best source for that description of what and who you know is your email inbox.

There's massive value to be unlocked in companies, and in our everyday lives from just being able to connect people. You can already see glimmerings of this in social networks like Facebook and LinkedIn which offer computer-aided ways to strengthen and navigate your relationships. Using the implicit data from our existing communications (like I'm experimenting with at twitter.mailana.com) we'll be able to build far more effective tools to link people together.

Photo by Junjan

One of the hardest things to do when you join Twitter is figuring out who to follow. Once you've added your immediate friends a good next step is people who live locally. There's a variety of ways to do this – first give http://twitter.mailana.com/find.php a try. Enter the name of your city or town, and it will bring up a list of everybody who's recently Twittered from there. It's a stepping stone to graphing the social networks for every location, so stay tuned.

Another good way to find local people is to use Twitter's built-in search. Enter your location in the 'Near this place' field and it will give you recent messages from people nearby.

To get a list of the most active and followed users in your neighborhood, go to http://twitter.grader.com/location and search for your home town. You'll see a ranked list, it's good to pick up on any local net celebrities you might not know about.

Photo by Nebarnix

Twitter's Search API is fantastic, but unlike the UI version they don't support using the name of a place to limit your results, only latitude and longitude. I couldn't ask my users to type in coordinates for searches, so I needed some way to translate the names into the form Twitter wants.

The process of converting from human names into coordinates is known as geocoding, and luckily there's some great free web APIs that will do the job for you. I chose Yahoo's GeoPlanet, they allow up 50,000 requests a month, and Tyler Hall has a great PHP wrapper ready to use.

I couldn't find a good example of actually using GeoPlanet in PHP, so I decided to kill two birds with one stone and publish my code as sample code for geocoding, and offer an off-the-shelf solution for anyone else who needs 'near' for Twitter searching. You can download geosearch.php
, and it's included below.

<?php

// Example demonstrating how to use Yahoo's Geoplanet API (via Tyler Hall's PHP wrapper)

// to emulate the 'near' parameter that's available in the Twitter search UI, but not

// through the API.

//

// To install it:

// 1) Put this file and class.geoplanet.php on your server

// 2) Get a free application ID from Yahoo – http://developer.yahoo.com/geo/

// 3) Copy the ID where this file has <Put your own app id in here>

// 4) Remove the die() statement just above it

//

// To use it:

// Call the URL of the file as you would http://search.twitter.com/search.atom

// All the normal supported arguments are passed through to Twitter's API

// If 'near=' is specified, then the place name string is passed to Geoplanet

// The latitude and longitude of the result are passed as 'geocode=' to Twitter

// If you also specify 'radius=' then that is passed, otherwise '25km' is used

//

// Gotchas:

// – It uses a redirect header, so for command-line curl you'll need -L to follow

// – There's zero error or exception handling

//

// Example:

// http://yourserver.com/geosearch.php?q=biking&near=boulder&radius=15km

require_once("class.geoplanet.php");

// If the named parameter was present on the input URL, append it to $url

function copy_url_input($name, $url)

{

if (isset($_GET[$name]))

$url .= "$name=".urlencode($_GET[$name])."&";

return $url;

}

$searchurl = "http://search.twitter.com/search.atom?&quot;;

// If the near parameter is present, then call Geoplanet to turn it into lat,long

if (isset($_GET['near']))

{

$placename = $_GET['near'];

die('You need to put your own app id here and then remove this statement');

$geoplanet = new GeoPlanet('<Put your own app id in here>');

$placelist = $geoplanet->getPlaces($placename);

$topplace = $placelist[0];

$centroid = $topplace['centroid'];

$lat = $centroid['lat'];

$lng = $centroid['lng'];

// The radius parameter is optional. If it's not present then default to 25km

if (isset($_GET['radius']))

$radius = $_GET['radius'];

else

$radius = '25km';

$searchurl .= "geocode=".urlencode("$lat,$lng,$radius")."&";

}

else

{

$searchurl = copy_url_input('geocode', $searchurl);

}

$searchurl = copy_url_input('q', $searchurl);

$searchurl = copy_url_input('lang', $searchurl);

$searchurl = copy_url_input('rpp', $searchurl);

$searchurl = copy_url_input('page', $searchurl);

$searchurl = copy_url_input('sinceid', $searchurl);

$searchurl = copy_url_input('show_user', $searchurl);

// Request a redirect to the URL with the lat,long added

header("Location: ".$searchurl);

?>

Photo by Suchitra Prints

http://twitter.mailana.com/ has been in the wild for a week now, and the best part has been hearing from users. I'm only human, so of course I love it when people tell me how well it's working; but as a developer the best comments are the complaints. Here's an example of why they're so useful.

I have a search filter set up for Mailana so any Twitter messages mentioning it show up in my RSS reader. On Saturday evening I spotted this from @mayaREguru

That Mailana thing is way wrong wrong wrong and weird. Makes no sense at all

The first thing that caught my attention was that she wasn't just hitting some minor glitches, it sounded like it was completely broken for her. That made my ears perk up, because if others are hitting similar problems it will be ruining a lot of people's experiences. Most of those people won't complain, so a single comment likely represents tens or hundreds of unimpressed potential users.

I got in touch with Maya and she gave me some more details. The whole BFF list was showing up incorrectly, with a lot of people she frequently talked to missing, and other people she'd only exchanged a few messages with listed instead. It's actually a lot easier to debug a massive problem like that than something more subtle, so I set to work figuring out what was going wrong.

Mailana works by mapping conversations. To know what conversations someone's had in Twitter, I need to import all the status updates for both sides of the conversation. That means for any user, there might be hundreds of other people they've sent @ messages to, and I need to download all their status updates to discover if those other people replied. When a new graph needs to be generated for a person, I first grab all their own messages, and then import all the messages of everyone they've sent messages to.

When I looked at Maya's information in the database, I could see there were a lot of people she'd sent messages to who hadn't been imported, so as far as the system was concerned she'd not had conversations with them, since it never saw any replies. I ignore those one-sided exchanges, since I'm looking for real relationships where there's two-way communication, so all those unimported people weren't showing up in the list or the graph.

Now I knew the immediate cause of the problem, but I had to work out why those people hadn't been imported. I won't go into the gory details, but after some debugging it turned out a crucial bit of code was looking at the wrong piece of information when figuring out the list of people to grab messages for.

It turned out my initial guess was correct, the same problem would have affected a lot of people's profiles. So by fixing Maya's graph I solved it for them too.

The issue never showed up in my own testing and I'd still be silently losing potential users still if I hadn't heard from her. That's why bug reports are like gold dust.

When I was working in the game industry writing tools for artists, I'd literally be sitting back-to-back with my users. Being that close to my customers made me feel their pain (and hear their cursing). That guidance helped me produce the programs I'm proudest of. Knowing they could turn around and slap me on the back of the head when it crashed concentrated my mind wonderfully!

Thanks to @mayaREguru, @robertshippey, @aleferreira and everyone else who's told me about things going wrong. Keep the complaints coming, it's really helping me improve Mailana!

Photo by OneEighteen

I wanted to add a tag cloud of all the terms people had used in their exchanges, to give an idea of what any two people were talking about. Normally I'd just query my Mailana system directly for the messages' content, but since my server is pretty busy importing new people and generating graphs, I thought I'd try Twitter's search API on the client side, using JSON to load the results.

I'm now using this on http://twitter.mailana.com/ to generate the clouds you see on the bottom left when you look at somebody's profile, but to make it more understandable I've created a standalone example at http://web.mailana.com/labs/twitcloud/

It works by constructing a new script element within the document pulling from the JSON search location. You can supply a callback function name, and that gets called back with the results of the search. View the source to understand what's going on, you're free to reuse any of this code in your own projects with no restrictions.

The biggest drawback is that the search API has been somewhat unreliable for me today, sometimes returning server 500 errors repeatedly, which results in a garbled callback. Despite that I'm still blown away by how much Twitter exposes through their API, it's a developer's dream.

Photo by funtik.cat

I don't agree with everything that Alex Payne writes in his latest blog post (I think there's a lot of potential in PostBox and Gmail's lack of inline messages is a major barrier for me) but he's spot on that email is caught in a strange time warp. There have been very few real advances in years, as the Foundry guys put it did Darwin skip over email?

A big part of the problem is that email is a closed silo. Any tiny startup can crawl the web, try something different with the data and then show it to potential users. The barriers for mail are much higher, you either need to persuade people to migrate to an entirely new email account to use your service, or write a desktop app with all the overhead that entails. I've been heavily focused on finding chinks in the armor surrounding all the email silos, since I think that's the only way we're going to see real innovation.

Photo by Eye2Eye

Our inboxes contain deeply private messages, so I'm used to hearing an initial reaction like this from Pascal Van Hecke

Watching http://web.mailana.com/demo/ don't understand anyone can believe employees will ever let their employers analyse all of their email

I need to create a full video to address why that first impression is wrong, but here's the short version:

Employees have complete control. The system suggests a profile for every person, but they're free to alter it, or even not publish it at all. There's a white-list of expertise keywords so all the suggestions will be safe, and nothing is revealed but your profile.

Employees benefit. I'm passionate about building this thanks to my experiences at the Apple coal face. Ask anyone working in a large company, they desperately need better ways of finding expertise. My goal is to save some unneeded late nights and hair loss and help people get things done faster.

Big Brother doesn't pay. There is a small market out there for business intelligence and  'compliance' monitoring of email, but very few companies want to alienate their employees by making them feel spied on. The big opportunity is in building a tool that is eagerly adopted by users, and that means building something they can trust to protect their privacy.

Similar projects were popular. I've spoken directly to a number of users of both Microsoft Knowledge Network and Tacit, earlier attempts based on similar ideas. Everyone who'd used them was very happy with the results.

I know I'll have to fight hard to win people over, but this really is an innovation that can make the world better, not another step towards an Orwellian nightmare. As @ev said at TED "when you give people better ways to share information, great things happen".

I released Mailana running on public Twitter data today! Tim O'Reilly kindly retweeted a link to it, so it's had a good load test. Fingers-crossed it's handling it so far.

If you're a new reader, here's a bit of background. My names's Pete Warden, I'm the founder of Mailana, and up until 6 months ago I was an engineer at Apple. I can't imagine a more impressive corporation to work for, but I knew there had to be better ways to collaborate within big companies. I spent a lot of time tracking down internal experts and the right people to talk to at external organizations by word of mouth. There was a lot of wasted work simply because employees in one department had no way of telling that their problem had already been solved by another team.

One day, I realized that just by analyzing my inbox, I could create a pretty decent profile of what projects and areas I was involved in, by looking for how often keywords show up, and who I talk to both within the company and at external customers and suppliers. Being able to easily build these expertise and contact profiles would let companies build directories to allow employees to easily find the resources they need. The biggest problem is ensuring that the privacy of people's email is respected, so the system I designed only publishes profiles after each employee has reviewed and edited their own.

You can see an example of how the system works here:
http://web.mailana.com/demo/

To build these applications, I designed a cloud-based system that imports from Exchange, Gmail accounts via IMAP, Outlook PST files and Twitter, offers a simple REST API to the stored data, and allows you to build apps as web widgets that can be deployed through the browser, on Sharepoint and as native menu entries and windows within Outlook.

Since there's not much public email out there, I decided to demonstrate what it's capable of using the massive number of public messages available on Twitter. My system internally processes a standard XML format, so I just wrote an importer that converted tweets into the right form, the rest of the system is identical to the Exchange version. I tidied up the web app that displays your immediate contacts, and had a Twitter social graph viewer up and running.

I'm hopeful that http://twitter.mailana.com/ will demonstrate how much interesting information there is sitting unused in your personal communication data. I think there's some amazing opportunities to build really useful tools, as long as we can design systems that preserve privacy. Imagine a version of LinkedIn that knew how close you were to all your contacts! Let me know what you think, and if your organization is interested in giving Mailana a try.