How to emulate near in the Twitter Search API using GeoPlanet

Atlassquashed

Photo by Nebarnix

Twitter's Search API is fantastic, but unlike the UI version they don't support using the name of a place to limit your results, only latitude and longitude. I couldn't ask my users to type in coordinates for searches, so I needed some way to translate the names into the form Twitter wants.

The process of converting from human names into coordinates is known as geocoding, and luckily there's some great free web APIs that will do the job for you. I chose Yahoo's GeoPlanet, they allow up 50,000 requests a month, and Tyler Hall has a great PHP wrapper ready to use.

I couldn't find a good example of actually using GeoPlanet in PHP, so I decided to kill two birds with one stone and publish my code as sample code for geocoding, and offer an off-the-shelf solution for anyone else who needs 'near' for Twitter searching. You can download geosearch.php
, and it's included below.

<?php

// Example demonstrating how to use Yahoo's Geoplanet API (via Tyler Hall's PHP wrapper)

// to emulate the 'near' parameter that's available in the Twitter search UI, but not

// through the API.

//

// To install it:

// 1) Put this file and class.geoplanet.php on your server

// 2) Get a free application ID from Yahoo – http://developer.yahoo.com/geo/

// 3) Copy the ID where this file has <Put your own app id in here>

// 4) Remove the die() statement just above it

//

// To use it:

// Call the URL of the file as you would http://search.twitter.com/search.atom

// All the normal supported arguments are passed through to Twitter's API

// If 'near=' is specified, then the place name string is passed to Geoplanet

// The latitude and longitude of the result are passed as 'geocode=' to Twitter

// If you also specify 'radius=' then that is passed, otherwise '25km' is used

//

// Gotchas:

// – It uses a redirect header, so for command-line curl you'll need -L to follow

// – There's zero error or exception handling

//

// Example:

// http://yourserver.com/geosearch.php?q=biking&near=boulder&radius=15km

require_once("class.geoplanet.php");

// If the named parameter was present on the input URL, append it to $url

function copy_url_input($name, $url)

{

if (isset($_GET[$name]))

$url .= "$name=".urlencode($_GET[$name])."&";

return $url;

}

$searchurl = "http://search.twitter.com/search.atom?&quot;;

// If the near parameter is present, then call Geoplanet to turn it into lat,long

if (isset($_GET['near']))

{

$placename = $_GET['near'];

die('You need to put your own app id here and then remove this statement');

$geoplanet = new GeoPlanet('<Put your own app id in here>');

$placelist = $geoplanet->getPlaces($placename);

$topplace = $placelist[0];

$centroid = $topplace['centroid'];

$lat = $centroid['lat'];

$lng = $centroid['lng'];

// The radius parameter is optional. If it's not present then default to 25km

if (isset($_GET['radius']))

$radius = $_GET['radius'];

else

$radius = '25km';

$searchurl .= "geocode=".urlencode("$lat,$lng,$radius")."&";

}

else

{

$searchurl = copy_url_input('geocode', $searchurl);

}

$searchurl = copy_url_input('q', $searchurl);

$searchurl = copy_url_input('lang', $searchurl);

$searchurl = copy_url_input('rpp', $searchurl);

$searchurl = copy_url_input('page', $searchurl);

$searchurl = copy_url_input('sinceid', $searchurl);

$searchurl = copy_url_input('show_user', $searchurl);

// Request a redirect to the URL with the lat,long added

header("Location: ".$searchurl);

?>

Why I love complaints

Hearttunnel

Photo by Suchitra Prints

http://twitter.mailana.com/ has been in the wild for a week now, and the best part has been hearing from users. I'm only human, so of course I love it when people tell me how well it's working; but as a developer the best comments are the complaints. Here's an example of why they're so useful.

I have a search filter set up for Mailana so any Twitter messages mentioning it show up in my RSS reader. On Saturday evening I spotted this from @mayaREguru

That Mailana thing is way wrong wrong wrong and weird. Makes no sense at all

The first thing that caught my attention was that she wasn't just hitting some minor glitches, it sounded like it was completely broken for her. That made my ears perk up, because if others are hitting similar problems it will be ruining a lot of people's experiences. Most of those people won't complain, so a single comment likely represents tens or hundreds of unimpressed potential users.

I got in touch with Maya and she gave me some more details. The whole BFF list was showing up incorrectly, with a lot of people she frequently talked to missing, and other people she'd only exchanged a few messages with listed instead. It's actually a lot easier to debug a massive problem like that than something more subtle, so I set to work figuring out what was going wrong.

Mailana works by mapping conversations. To know what conversations someone's had in Twitter, I need to import all the status updates for both sides of the conversation. That means for any user, there might be hundreds of other people they've sent @ messages to, and I need to download all their status updates to discover if those other people replied. When a new graph needs to be generated for a person, I first grab all their own messages, and then import all the messages of everyone they've sent messages to.

When I looked at Maya's information in the database, I could see there were a lot of people she'd sent messages to who hadn't been imported, so as far as the system was concerned she'd not had conversations with them, since it never saw any replies. I ignore those one-sided exchanges, since I'm looking for real relationships where there's two-way communication, so all those unimported people weren't showing up in the list or the graph.

Now I knew the immediate cause of the problem, but I had to work out why those people hadn't been imported. I won't go into the gory details, but after some debugging it turned out a crucial bit of code was looking at the wrong piece of information when figuring out the list of people to grab messages for.

It turned out my initial guess was correct, the same problem would have affected a lot of people's profiles. So by fixing Maya's graph I solved it for them too.

The issue never showed up in my own testing and I'd still be silently losing potential users still if I hadn't heard from her. That's why bug reports are like gold dust.

When I was working in the game industry writing tools for artists, I'd literally be sitting back-to-back with my users. Being that close to my customers made me feel their pain (and hear their cursing). That guidance helped me produce the programs I'm proudest of. Knowing they could turn around and slap me on the back of the head when it crashed concentrated my mind wonderfully!

Thanks to @mayaREguru, @robertshippey, @aleferreira and everyone else who's told me about things going wrong. Keep the complaints coming, it's really helping me improve Mailana!

A Twitter JSON Search API Example

Cloudscape

Photo by OneEighteen

I wanted to add a tag cloud of all the terms people had used in their exchanges, to give an idea of what any two people were talking about. Normally I'd just query my Mailana system directly for the messages' content, but since my server is pretty busy importing new people and generating graphs, I thought I'd try Twitter's search API on the client side, using JSON to load the results.

I'm now using this on http://twitter.mailana.com/ to generate the clouds you see on the bottom left when you look at somebody's profile, but to make it more understandable I've created a standalone example at http://web.mailana.com/labs/twitcloud/

It works by constructing a new script element within the document pulling from the JSON search location. You can supply a callback function name, and that gets called back with the results of the search. View the source to understand what's going on, you're free to reuse any of this code in your own projects with no restrictions.

The biggest drawback is that the search API has been somewhat unreliable for me today, sometimes returning server 500 errors repeatedly, which results in a garbled callback. Despite that I'm still blown away by how much Twitter exposes through their API, it's a developer's dream.

Email isn’t there yet

Messageinabottle

Photo by funtik.cat

I don't agree with everything that Alex Payne writes in his latest blog post (I think there's a lot of potential in PostBox and Gmail's lack of inline messages is a major barrier for me) but he's spot on that email is caught in a strange time warp. There have been very few real advances in years, as the Foundry guys put it did Darwin skip over email?

A big part of the problem is that email is a closed silo. Any tiny startup can crawl the web, try something different with the data and then show it to potential users. The barriers for mail are much higher, you either need to persuade people to migrate to an entirely new email account to use your service, or write a desktop app with all the overhead that entails. I've been heavily focused on finding chinks in the armor surrounding all the email silos, since I think that's the only way we're going to see real innovation.

Will anyone use Mailana?

Conversation

Photo by Eye2Eye

Our inboxes contain deeply private messages, so I'm used to hearing an initial reaction like this from Pascal Van Hecke

Watching http://web.mailana.com/demo/ don't understand anyone can believe employees will ever let their employers analyse all of their email

I need to create a full video to address why that first impression is wrong, but here's the short version:

Employees have complete control. The system suggests a profile for every person, but they're free to alter it, or even not publish it at all. There's a white-list of expertise keywords so all the suggestions will be safe, and nothing is revealed but your profile.

Employees benefit. I'm passionate about building this thanks to my experiences at the Apple coal face. Ask anyone working in a large company, they desperately need better ways of finding expertise. My goal is to save some unneeded late nights and hair loss and help people get things done faster.

Big Brother doesn't pay. There is a small market out there for business intelligence and  'compliance' monitoring of email, but very few companies want to alienate their employees by making them feel spied on. The big opportunity is in building a tool that is eagerly adopted by users, and that means building something they can trust to protect their privacy.

Similar projects were popular. I've spoken directly to a number of users of both Microsoft Knowledge Network and Tacit, earlier attempts based on similar ideas. Everyone who'd used them was very happy with the results.

I know I'll have to fight hard to win people over, but this really is an innovation that can make the world better, not another step towards an Orwellian nightmare. As @ev said at TED "when you give people better ways to share information, great things happen".

See your top friends on Twitter

Twitterfriendslogo

I released Mailana running on public Twitter data today! Tim O'Reilly kindly retweeted a link to it, so it's had a good load test. Fingers-crossed it's handling it so far.

If you're a new reader, here's a bit of background. My names's Pete Warden, I'm the founder of Mailana, and up until 6 months ago I was an engineer at Apple. I can't imagine a more impressive corporation to work for, but I knew there had to be better ways to collaborate within big companies. I spent a lot of time tracking down internal experts and the right people to talk to at external organizations by word of mouth. There was a lot of wasted work simply because employees in one department had no way of telling that their problem had already been solved by another team.

One day, I realized that just by analyzing my inbox, I could create a pretty decent profile of what projects and areas I was involved in, by looking for how often keywords show up, and who I talk to both within the company and at external customers and suppliers. Being able to easily build these expertise and contact profiles would let companies build directories to allow employees to easily find the resources they need. The biggest problem is ensuring that the privacy of people's email is respected, so the system I designed only publishes profiles after each employee has reviewed and edited their own.

You can see an example of how the system works here:
http://web.mailana.com/demo/

To build these applications, I designed a cloud-based system that imports from Exchange, Gmail accounts via IMAP, Outlook PST files and Twitter, offers a simple REST API to the stored data, and allows you to build apps as web widgets that can be deployed through the browser, on Sharepoint and as native menu entries and windows within Outlook.

Since there's not much public email out there, I decided to demonstrate what it's capable of using the massive number of public messages available on Twitter. My system internally processes a standard XML format, so I just wrote an importer that converted tweets into the right form, the rest of the system is identical to the Exchange version. I tidied up the web app that displays your immediate contacts, and had a Twitter social graph viewer up and running.

I'm hopeful that http://twitter.mailana.com/ will demonstrate how much interesting information there is sitting unused in your personal communication data. I think there's some amazing opportunities to build really useful tools, as long as we can design systems that preserve privacy. Imagine a version of LinkedIn that knew how close you were to all your contacts! Let me know what you think, and if your organization is interested in giving Mailana a try.

Get customers

Customerservice

Photo by Matt Watts

The only way to figure out what your users want from your software is to release it and get feedback. As an engineer, this is something I constantly struggle with, because at any given time the actual product is only a pale shadow of what I know it could be if I had just one more week! How can I possibly release it in this state?!

Having a release deadline forces you to prioritize and fix the boring problems you know the user will hit, rather than doing something more fun and arcane. Once it's released, you'll also rapidly discover that users don't care about the things you expect, and drive the product in completely unexpected directions.

A couple of articles this week were good reminders to keep focused on getting releases into the wild. Tom Evslin blogged his advice for entrepreneurs, drawing on Jeff Jarvis's new book What Would Google Do? When Google News was about to be released, the programmers were warring over whether to put their resources into sorting by date, or location. They couldn't decide, so it was released with neither, and 300 of 305 emails from users asked for date.

In a darker scenario, Eric Ries talks about a project he'd been involved with that waited too long to release. It was a well-funded, multi-year project, staffed with smart people with a clear plan that they executed on. I've seen a lot of these at engineering-driven companies, where a powerful figure with a technical background will spin tales of the wonderful software he can build, if he's only given enough resources. They always feel like the Apollo program, full of idealism and shooting for audacious goals. The trouble is, you don't find out if your goals match the customer's needs until the end. No matter how many market surveys and focus groups you run, you'll never get a clear idea of what people will actually pay for. Building it in one shot like that also means you've got a massive foundation of instant legacy code that makes it hard to adapt once you do hear from real users. That's exactly what happened to Eric's project, after pouring years and millions of dollars into the software it flopped on release and the company failed.

Have a strong vision of what you wanted to achieve and big hairy audacious goals, but stay on course by releasing early and often.