Photo by Chris JD

Between the shutdown of Ubermedia and the company's announcement that it will not allow any new third-party clients, the once-passionate love affair between Twitter and third-party developers has finally descended into arguing about whose turn it is to do the washing up. I've seen this happen before.

In my past life at Apple I was responsible for some third-party developer support, and that was the worst part of my job. Not because of the external people I was dealing with, I loved them all and still stay in touch. That was actually the problem, because I often had to be a complete asshole to them. That bug-fix that would save you days, weeks or months of engineering and support time and would be a quarter-day job for us? No, not going to happen. That API you were relying on? We're removing that, with no equivalent available. Oh, and we liked your idea so much, we're bringing out our own version next month. I actually managed to smuggle in a few changes, which helped me keep a shred of my self-respect, but it was spirit-breaking work. Here's what it taught me.

Don't Believe the Hype. Everybody in the engineering team was a fan of what third-party developers were doing with our APIs, and wanted to encourage and help them however they could. Especially in the early days, we sit around with developers and get excited about what was possible, getting their hopes up. Unfortunately our enthusiasm was no match for relentless schedules and bug-fix prioritization, so we always had radically less resources to give them than we'd hoped for. When engineering hopes collides with business priorities, revenue wins. Looking back, I feel most guilty about the way I let my own enthusiasm lead third-party developers on, and I'd imagine the Twitter engineers feel something of that too.

It's Not Personal. When your dreams and livelihood are on the line, it's hard not to feel like it's the fault of the people you're dealing with. Why won't they listen, it's so obvious it makes sense? If you could just get somebody sensible on the phone, they'd understand. You can see this on the Twitter development list as people ask for Ryan to add another point of contact, presumably in the hope that the disliked changes are a misunderstanding. In fact, most external actions like this tend to be very deliberate expressions of corporate policy, even though the messages are often sugar-coated to minimize the damage. Whoever was in that position would be forced to do the same thing by their bosses, who in turn are under pressure from investors. External developers have almost no leverage in power struggles over corporate policies, and that doesn't change depending on who the front-man is.

Never Become a Sharecropper. Take a long, hard look at the power balance behind any long-term business relationship you enter into. Imagine you become wildly successful. How much of that success can you keep if you're tied to one platform, at the mercy of arbitrary changes in the terms of service? What's the track record of the provider?

Sometimes it makes sense – Microsoft were actually awesome to work with as a third-party developer, often-times to the detriment of the end consumer who was forced to turn to external tools for things that really should have been in the OS. For ISVs this was great though, you knew Microsoft would generally avoid competing directly with you, and would likely offer to buy you out at least if they were going to tread on your turf.

Open Wins (Eventually). We go through cycles of reincarnation, as a technological change sweeps the landscape and leaves a few companies with seemingly-unassailable strangle-holds on the market. The beauty of computing that we keep adding new layers of indirection, and sooner or later we route around the obstruction. Microsoft looked unbeatable, now I'd bet the majority of Windows installations are on virtual machines and apps are increasingly running in a browser. Nobody beat them at their game, we all just moved to a new playing field. Facebook is the current king, they've won the social network contest convincingly. But unless they anticipate everything that people want in the future, and deliver it all, there will be applications flourishing outside of their walled garden, and that's where the future will come from.

If you keep that in mind as a developer, you'll avoid tempting dead-ends that offer great initial distribution, but don't give you any real control over the data or platform. Open source projects can be a massive pain in the arse, they're messy, incomplete and often have trouble reaching users, but they're the only foundation you can build anything lasting on.

Photo by Brenda Gottssabend

I've been volunteering on the Ushahidi Swift pipeline, and it's been fascinating. The goal is to turn massive raw streams of tweets, text messages and news stories into useful information, something that's especially relevant looking at all of the recent turmoil around the world. Pulling data from unstructured text is one of the toughest and most interesting problems around, and Swift is a great platform to learn and test new approaches.

So I was pleased to hear from Jon Gosier that they're looking to hire some more engineers. The pay's not great compared to West Coast standards, but they're open to hiring globally, so it makes more sense for people in an area with a lower cost of living. They're also happy to accommodate side projects and offer a lot of flexibility as long as the work gets done. If you're looking a job on a great team working on important problems, this could be a good chance to gain experience in some very useful areas. Here's the full job posting.

[Update – I wrote this at the end of a long and frustrating day, and got the tone completely wrong, I was way too grumpy. Normally I wouldn't revise, but since it ended up on HN, I'll add this note. I'm still living in awe that I can rent a hundred machine cluster for $10 an hour, for me that's better than a jet-pack]

Today I needed to create a Rackspace account, so I could help out a non-profit I'm supporting. I've long been intrigued by the idea of a cloud host with support as a key feature, especially after the poor experience I had when my Amazon load balancer died. That left me glad to have an excuse to try out Rackspace, but after going through signup I was left distinctly unimpressed. There wasn't anything major, but there were several aspects to the process that were jarring.

Password

They require the password to have a mixture of upper and lower-case letters and numbers. This is a cop-out, they should be testing for general password strength rather than inflicting arbitrary rules like this. It's not a massive coding task, and it's also odd to see them passing the plain text passwords back to the server for checking, rather than doing it client side. The page is https (though with warnings on Chrome about untrusted content) so it's not a major flaw, but it's poor fit-and-finish. It did fill me with fear that they might be storing my password as plain text though, especially when I discovered they send root passwords for your boxes through email!

Checkbox

You also have to pick a product on the account creation page. This is odd, but what was worse was that it's at the top of the page and gets unchecked when your password fails their rules. This meant I still couldn't submit the form after I'd changed the password to comply, and there was no feedback as to why the submission was failing. Again, a minor but irritating missing detail.

Verification

After all that, one of their employees has to give me a phone call before I can start creating servers. It felt very old-fashioned, and meant that the self-serve aspect was gone. Amazon does something similar, but has it all automated, giving me a pin I can put in, a much smoother experience. I had to stop progress on exploring the service and wait for a call, and of course when it came I was in a meeting, so I couldn't answer it. Later in the day, I called back the number they emailed me, got the general customer service line, had to sit through two repetitions of options, none of which appeared to apply, until I finally got a real person. He took some details, and had to forward me to an on-boarding specialist. I waited for a few minutes on hold, and then he had me confirm some basic details about my credit card number and address.

I was waiting for the reason they did this as a call with a human. I wasn't even calling from the same number I'd given them, so caller ID wasn't giving them additional confirmation. I thought they'd try to wow me with a sales pitch at least, or sweep me off my feet with an insistence on answering any questions I might have, but it was just a polite data-entry conversation.

Shared Server Images

This one's more of a missing feature than a user experience problem, but it contributed to my grumpiness. My goal with the non-profit is to set them up with a server containing some useful software, a mixture of open source tools built by me and others. The installation process for all of these packages is a multi-page document, taking a fair amount of time. On Amazon, I'm able to do this once and build a shared AMI, making it public so anyone can use the system. It looks like on Rackspace I'll have to go through the installation process for every account that wants to use the system, since they only support server images within a single account. It's clear from comments on that announcement that I'm not the only one looking for this, but there's still no ETA on the feature.

I'm really hoping I'll see the advantages of Rackspace's approach now I'm signed up. If it was a phone company or bank, I'd just sigh and move on, but I had higher hopes for a Zappos-like experience. My first three complaints could all be solved with some fairly simple tweaks to their processes, a bit of Javascript and some Twilio. As it stands, though Amazon's not perfect, their sign-up process actually gave me a lot better user experience.

Photo by Cat O

Egypt Influence Network – A network graph of Twitter users that actually tells a story. I do wish there was more discussion of the method and data used to generate it though. We all need reproducibility and peer review to strengthen our arguments.

CKAN – the Data Hub – A rich curated collection of data sets and APIs, with hopes to become the CPAN for information.

Weatherspark – Very effective interactive presentation of weather data. I especially like the faded background to the temperature graph subtly showing the mean, min and max values over time.

Spreadsheet Scraper – Simple but nice Chrome extension for pulling tabular data from web pages. Increasingly the client side is the only place to access third-party information, as robots.txt and API terms-of-service become more and more restrictive.

NoSQL @ NetFlix – It’s very helpful that Sid has done so much work explaining NetFlix’s experiences porting applications to NoSQL. It’s often hard to convince companies with existing traditional infrastructures that switching is an option worth considering, these sort of case studies really help.

Photo by Diane Cordell

The Importance of Making Things – US Science fairs are reported to be in decline, which is obviously deeply worrying for the future. It's easy to forget what a lifeline these sort of extra-curricular activities can be for individual kids though. Retreating to the library was a bright spot in my school day, Ta-Nehesi found his open space and pride with his science projects.

North Korea Uncovered – I'm late to discover this, but a fascinating attempt to map the Hermit Kingdom using public satellite imagery.

Making a Difference in the Developing World – I've found myself getting more and more involved in open source projects aimed at poor countries. I'm all too aware of the high failure rate of traditional aid and my weird position as a westerner with very little understanding of the end-users actual requirements. To compensate for that, I try to use what people on the ground are actually doing as a guide. With that in mind, I was pleased to see how useful Mendeley's tool seems to be in the context of African agricultural research. I don't know how that lesson generalizes, but it might be that online collaboration tools are even more important in places where professional organizations and institutions are still developing.

MetaQuerier – The research project that Cazoodle grew out of. I love this because it's an academic formulation of the approach used by most successful commercial projects to analyze web data. Most academic research in this area seems to fall into to the deadly tar-pit of Linked Data, which is brought to you by the same people who didn't bring you the Semantic Web, and suffers the same drawbacks of requiring mass publisher adoption before delivering benefits.

Building Interfaces for Data Engines – Matthew Hurst is one of the pioneers of analyzing web-scale data, so I've been pleased to see him blogging so prolifically recently. He's also just launched a data exploration tool called d8taplex, and this post explores some of the different approaches that other data set providers take to interacting with the information. Fascinating stuff, and I'm not just saying that because he's looking over my book, honest.

Photo by Eric Gjerde

Yesterday I laid out why I'm not looking for investment. I'm comfortable in my choice, but it wasn't easy. The investment process brings a lot more than cash; here's what I'm losing by avoiding it.

Discipline

Convincing someone to give you money forces you to get very specific, and flesh out your business plan. Defending it against hostile questions makes it stronger. Once you've raise money, having somebody who remembers your promises breathing down your neck keeps you focused. My biggest weakness is that I'm too prone to stay in my comfort zone, working on technical experiments and neglecting everything else you need to build a business. I try to self-police that tendency, but there's nothing like having an outsider's view to keep you on track.

Time

I spend a lot of my week dealing with paperwork, doing website housekeeping, taxes, system administration, arranging travel and meetings, and all the other tasks that could be handled by somebody else if I had funding. I'd have to spend time managing investors instead, but at least the rest of my schedule could be devoted to customer support and product development.

Advice and Influence

Most investors want to convince you they're 'smart money', but from what I've experienced, not many actually bring much to the table beyond dollars. When they do though, it can make a massive difference. They can give advice that saves months of development time, and set up deals that bring in crucial revenue. The key is identifying the people who have the will and the ability to help like that. The only way to do this is by looking for a track record, and knowing entrepreneurs you trust that they've worked with before. A well-known brand is definitely not enough. In my experience the superstar firms are less eager to roll up their sleeves, they're less hungry, though the clout of their name as a credential can make up for a lot of that when it comes to making deals.

Photo by Matt Katzenberger

The short answer? I don't have a good enough idea.

Living in San Francisco, in the current angel investment bubble, with a (tiny-but-nice) acquisition by Apple under my belt, and big plans to change the world, I sometimes feel like a freak of nature because I'm not hunting for funding. Why did I make that decision?

I spent the first year after I left Apple aiming to build a VC-backed company. I had a plausible story, built a complete prototype, deployed pilot installations, and sat in numerous boardrooms pitching partners. The trouble was, it was a terrible idea. I was following in the footsteps of Tacit, VisiblePath and multiple other enterprise email analysis failures. I had no comprehension of the torture of the enterprise sales cycle. The idea of analyzing email fundamentally creeped people out, no matter how strongly I protected privacy. In the classical model of how the process works, I would have got all this feedback from the VCs I pitched to. Aside from being pointed towards Tacit by one partner, I learned everything by painful trial-and-error. VCs loved the idea, they were deluged with email messages and made their livings through their personal connections.

The lesson I drew from that was that I was an idiot. I should be listening to customers and discount what I heard from investors. Get customer traction, revenue, and ideally a profit, and the opportunity would sell itself to potential funders. Obviously that also cut down the range of opportunities I could chase, but what I'd discovered in my unfunded year was that it's possible to build a lot with very little cash. By building fully-functioning prototypes, I could learn what users responded to, as well as what was technically possible.

The trouble is, after a couple of years of operating like this, I'm addicted. I've had many failures, along with a few partial successes, but I'm become so much better at understanding what people need, what's possible, and executing a solution. My failures are becoming faster and far more interesting. I may be living on Ramen noodles, but it's an amazing opportunity to learn. Just the process of fund-raising itself would take a massive amount of time away from that, and then explaining every month where their money's going, why I'm fishing with strawberries, and planning for the next round would take even more. 

That's still a worthwhile use of time if the funding is going towards rocket boosters to strap onto a killer idea. I don't think it's lack of confidence that's telling me none of the approaches I'm working on has crossed the threshold where it's worth it though. There's funding available, I just don't think the strings attached make up for the help. Better to live frugally, consult when I have to, and focus on learning. The more progress I can make, the more I can learn before I need to buckle in to the rocket sled, the better. I'm looking forward to honing my ideas until I can build the real business I dream of, whether that's fully boot-strapped or with the help of investors.

Word cloud

I was surprised to find that there were no transcripts available of Gadhafi's long speech on Tuesday (February 22nd), so after looking into transcription services I decided to bite the bullet and go through the whole 70 minutes myself. It was a deeply unpleasant experience, but it seems worthwhile to capture his words in an accessible form, since the world is facing so many tough choices in how to deal with him. I don't want to say he's crazy, that seems too easy, but the benevolent mask he obviously tries for keeps slipping as he nakedly threatens his 'people'.

https://docs.google.com/document/d/10dy5oLJY2QL7k2VuwKonUpSgCUX-_9ATQ-134Xka9fs/edit?hl=en#

It's a Google document that anyone can edit, so I'm hoping people who know the area much better than me will correct the many mistakes, especially with the place and people names. I also hope it does some good.

I generated a word cloud from the speech, to help me understand what the key obsessions were. People was a key verbal tic, as were 'youth', 'revolution' and 'rats'. Benghazi is obviously preying on his mind too, and he kept coming back to security. I can't claim to make sense of his speech at all, but it feels like the only avenue to understanding what's going on in his mind.

The books I love have their own unique souls, but I struggle to communicate what they are. I can describe the plot, or talk about the themes and the language, but there's an essence that's hard to get across. I find word visualizations a fresh way of capturing that flavor, but I've never been able to get the results I can picture in my minds eye. That drove me to build Wordlings, an experimental service for creating graphical word clouds. Here's what I came up with when I applied it to some of my favorite novels.

Kidnapped, by Robert Louis Stevenson

Carefully disguised as an adventure story, this is actually a beautifully-drawn portrait of friendship. The psychology of the protagonists' relationship veers between suspicion and reliance, anger and affection, in a way it's rare to read. Here's Stevenson's dissection of the tangled reaction to a dependent friend:

The thought of a separation ran always the stronger in my mind; and the more I approved of it, the more ashamed I grew of my approval. It would be a fine, handsome, generous thing, indeed, for Alan to turn round and say to me: "Go, I am in the most danger, and my company only increases yours." But for me to turn to the friend who certainly loved me, and say to him: "You are in great danger, I am in but little; your friendship is a burden; go, take your risks and bear your hardships alone—-" no,that was impossible; and even to think of it privily to myself, made my cheeks to burn.
And yet Alan had behaved like a child, and (what is worse) a treacherous child. Wheedling my money from me while I lay half-conscious was scarce better than theft; and yet here he was trudging by my side, without a penny to his name, and by what I could see, quite blithe to sponge upon the money he had driven me to beg. True, I was ready to share it with him; but it made me rage to see him count upon my readiness.

What jumps out from the word cloud is the centrality of Alan to the book. With his accidental companion David as the narrator, you can see how the interplay between them comes out in the frequent invocation of his name. You can also see how plain Stevenson's language is; "Back", "Down", "Man", "Like". It's why the book flows so easily, despite the depth of ideas it wrestles with. That made the ship that Alan and David were thrown together on feel like the right outline for the cloud, gliding through the water apparently effortlessly, while in fact relying on an exquisitely well-organized web of rigging. 

Pride and Prejudice, by Jane Austen

I'm a late-comer to Austen, only discovering this book when I was a teenager. I would read a few pages to my girlfriend every night, and the book's world became part of our private language. I'd always liked to think I was Darcy, I can certainly match his pig-headedness, but in my darker moments I feared the unwitting ridiculousness of Collins was a better match. The older I get, the more happy I'd be to just achieve Bingley's quiet decency and contentment. There's a lot to be said for nice, but dim.

It's fitting that names dominate the cloud, since the novel is like a pool shark's trick shot, as actions and reactions cascade in an intricate yet believable way. Austen's genius is the way she sketches characters who behave like actual people, but still contrives to arrange events to produce a satisfying outcome for them all. Her dark wit and cutting sarcasm make the happy endings palatable to a cheerful pessimist like me.

The shape to choose was obvious. The 1995 BBC mini-series left Jennifer Ehle indelibly as my personal picture of Lizzy, minxing from underneath her bonnet and curls. I was so hooked, I even forced my old D&D group to watch it, and even they were sold. As one of them said afterwards, "Man, that was the Star Wars of costume dramas!".

Oliver Twist, by Charles Dickens

Despite fears that he's better-known than read, I keep coming back to Dickens. His long-winded style forces me to switch gears, and the barrage of words is like a punch in the jaw compared to Austen's sweet smile while she sticks a stiletto in your ribs. It's effective though, and like reading David Foster Wallace, once you get accept the rhythm the chaos starts to make strange sense. He's unashamedly sentimental, which I appreciate, since I enjoy those sort of illusions even if I struggle to believe them.

The book revolves around power, authority and punishment, with the final chapter taking obvious pleasure in meting out lashes and rewards, sketching the future lives of even minor characters. I wonder if part of the wild success of his books was related to the audience's hunger for this sort of settling of scores? It's the same pleasure I get reading Sherlock Holmes, the relief of believing the world can make sense and is more than just one damn thing after another.

The authority is there in the words that pop out – 'gentleman', 'boy', 'old', 'Jew', 'young', 'lady'. Roles and positions are central to the action. The silhouette seemed to capture the spirit that Dickens was fighting against, self-satisfied and in charge, though I wish I'd found one with the true beadle's hat.

I'm finding this a rewarding experiment, if only for the way it's forcing me to look at my best-loved books in a new light. I'd love to hear about any you're able to create with Wordlings, and what you learned, so do let me know.

Photo by Dr Case

These days it doesn't make much sense to build a consumer site with its own private account system. If users sign in using Twitter or Facebook, you can have them easily spread the word on what they're doing in your application through tweets or wall posts. The old 'enter details'/'email confirmation'/'click link' process was a leaky funnel that lost potential users too, and was a pain to implement, since automated emails alone are an art in themselves.

I wanted to use this approach on one of my sites, but I couldn't find a good example. Especially in the Rails/Sinatra world there were plenty of building blocks available for doing the individual tasks, like authenticating through Twitter or Facebook, calling their APIs, and storing data, but nothing that showed how to use them together. So, because I needed to build it anyway and it would be a good Ruby learning exercise, I set out to create a minimal-but-complete template that handled user creation, authentication, custom data storage and external sharing. In the spirit of the classic 'Hello World' code examples, here's Hello Social World.

You can try a live demo at http://hellosocialworld.heroku.com/. It's built to do the bare minimum you need for a modern social site, letting users log in, edit some data they own on your site, and then share the results with their friends on Twitter or Facebook. The data in this case is just the user's favorite color, but imagine replacing that with whatever content they actually create on your site; blog posts, photo galleries, comments, etc. It's deliberately left with zero styling, so you can add your own view code.

I was really impressed by the elegance of Ruby and the Sinatra framework. Aside from the API keys, all the code's in a single 440 line file, and 120 of those lines are comments. This is possible because the gems handle most of the heavy lifting for me. It feels very productive to focus on my application logic instead of writing boilerplate code. There were moments of frustration when things went wrong, the black box nature of gems makes them tough to debug, but there seems to be a big enough community using them that they mostly just work.

I'll be using this template heavily myself over the next few weeks, which will hopefully give me ideas on improvements, but let me know if you have bug reports or suggestions. I'm looking forward to hearing people's thoughts on this approach to a turn-key social site too.