Why I hate client-side code (and the cloud will win)

CarcrashPhoto by Saiki

Most of my career's been spent on desktop or embedded systems code and I'm a relative newcomer to web programming. Despite the horrors of server-side development (debugger? ha!) it's so much faster to develop web services than traditional apps. The main reason is that I have control over far more of the environment when the code is running on my own box and I'm only relying on a client to display the UI. The testing matrix for Apple Motion was insane because it ran on the GPU, every piece of hardware behaved differently, and so as new graphics cards and machines came out the combinations we had to check exploded.

So, I have a lot of sympathy with Microsoft, and the Xobni folks doing client-side processing, but this novel-length KB article on troubleshooting Outlook crashes sums up why users are so happy with web apps, despite their limitations.

An implicit data bill of rights

Wethepeople
Photo by Vkx462

I've been lucky enough to spend some time with Ken Zolot this week, who's heavily involved with startups both through MIT and the Kaufmann foundation. He threw some fantastic papers in my direction, and one of the most interesting finds was a proposal by Alex Pentland on data privacy, what he calls a New Deal on Data. I've been wrestling with how to use implicit data on people's behavior in an ethical and honest way and Pentland's definition is really helpful.

He draws on English Common Law principles of possession, use and disposal, and applies them to data about ourselves, to match our intuitive feelings of ownership of information about ourselves.

1. You have a right to possess your data. Companies should adopt the role of a Swiss bank account for
your data. You open an account (anonymously, if possible), and you can remove your data whenever you’d like.

2. You, the data owner, must have full control over the use of your data. If you’re not happy with
the way a company uses your data, you can remove it. All of it. Everything must be opt-in, and not only clearly explained in plain language, but with regular reminders that you have the option to opt out.

3. You have a right to dispose or distribute your data. If you want to destroy it or remove it and redeploy it elsewhere, it is your call.

In practice these make some technical demands for the ability to export and delete information that few services provide. Try saving out your friend graph from Facebook without violating their terms-of-service!

This makes it a tough sell for corporations built around hoarding users' information as a proprietary asset. In the long-term though, the benefits of users sharing information widely will benefit services that don't lock in their users. You can already see that with Twitter's API; their lack of restrictions has led to applications that weren't even imagined before the data became available.

How can you measure influence?

Persuasion

Influence is the measure of your ability to persuade others to take an action. Micah of Lijit gave a barn-storming talk at Boulder NewTech last night, describing how they are starting to measure blogger's influence. It's not publicly released yet, but they're combining both raw audience figures and the user activity they measure through the Lijit widget, things like searches and clicks.

This is exciting to me because nobody's been able to use implicit data on people's behavior in a widespread way, because nobody's had access to a large enough set. I'm bullish on Lijit's prospects because they are in a unique position with hundreds of millions of user interactions across thousands of sites in their database (OneRiot are the only other company I can think of that's got access to more info through its browser add-on).

Lijit's measure is a big step forward, but did leave me with a couple of questions. Influence has to be defined around an action, but their measure seems to be positioned as a universal metric. Lolcats is a lot more likely to make me buy a t-shirt than the Sunlight Foundation, but lolcats has no influence on how I vote. If you pick a single influence number you can't capture that.

There's also the question of who you're influencing. A picture on lolcats will get a lot more pageviews than a post on Brad's blog, but a lot more influential people in the tech community will see the blog post. Google's PageRank tackles this by taking the influence of the people who link to a site into account to calculate its influence. That means a bunch of barely-read geocities (RIP) sites linking to you doesn't matter as much as a link from the New York Times. There's no equivalent way of compensating for the relevance of the users whose activity you're measuring. Having a single Steve Jobs viewing your pages is more influential than 1000 random teenagers.

I've some thoughts on fixing this, and actually started running PageRank on Twitter conversations to figure out who was most influential on the service, but had to put that on hold to focus on other work. I can't wait for Lijit to launch the rankings, despite all my niggling this should be a massive jump forward!

Are you taking market risk or technology risk?

Brokencasio
Photo by MHuang

If you're working in the pharmaceutical industry, your main risk is your new treatment won't work. There's a massive number of medical problems people are certain to pay money to solve, if you can create a drug that works.

In the toy business, it's completely different. Building that new Pet Rock or Cabbage Patch Doll is easy, but for very hard-to-predict reasons people may not like it. You may not be able to distribute or market it even if there are some who do.

Most startups lie on a spectrum between these two extremes of technology and market risk, but I've learnt it's crucial to understand what your mix is. People from a business background prefer market risk, because that's something they know how to measure and mitigate. Techies like me have a bias towards hard engineering problems that they know how to solve.

I started off thinking that Mailana's main risk was technology – it's really hard to integrate with Exchange, build Outlook plugins and analyze millions of emails in real time. There were all sorts of end-user problems that can be solved with the information derived from that, so once the system was built, customers would come. You can chuckle at my naivity, but I never understood that there were two separate risks. I put a lot more effort into coding than understanding the market, and then discovered there were all sorts of unexpected cultural issues around privacy that scuppered my first attempt when it was in front of customers

The beautiful thing about market risk is that you can take very simple steps to reduce it before you spend months coding. Build slideware and ask your potential customers if they'd buy a working version. Buy some relevant AdWords and point them at a dummy product page to see if anyone signs up for more information.

If you're reading this, you're not Pfizer and you do have a market risk. Take a long hard look at your business and see what you can do to reduce it.

How to make connections with people you don’t know

Stalking
Matt Van Horn from Digg gave a talk I wasn't expecting last night; the practical side of networking. The whole mission of Mailana is "You guys should talk", I love it when I can connect two people who can help each other. To make that happen, you have to be able to build bridges with strangers; Matt revealed his personal toolkit for reaching the right people.

Matt started off with the LOLCAT picture because if you're not used to networking it can feel creepy and exploitative. What I've realized, and Matt emphasized, is that you need to approach it as a way of helping other people, not just be a taker. It's a long-term project, not something you desperately turn to at the last minute when you need a job.

Having said that, the story of how Matt got to be business development manager at Digg is an example of how chutzpah pays off. He targeted Digg as a company he really wanted to work for, and queued for 2 hours at a trade show to get a business card from Jay Adelson, their CEO. After that he emailed him repeatedly trying to set up a meeting, as well as sending on relevant newspaper articles to the Digg offices. Then he guessed a couple of email addresses for their recruiter and CRO, and eventually landed an interview. They asked him to write a detailed description of the position he wanted to create in the company and how it would help Digg. Finally that landed him the job! Wouldn't you hire somebody who showed that much determination and resourcefulness?

Here's a few of the tips Matt gave out for getting in touch with people you want to talk to, but can't get a 'warm' introduction for:

Guess email addresses. Most companies have a fixed format, eg pete.warden@company.com, pwarden@company.com, pete@company.com. Figure it out from public examples or just guess and fire off a message.

Call at odd hours. Receptionists are usually only there 9-5, but most of us work before and after, so there's a good chance somebody helpful will pick up if you ring 7:00am to 9:00am or 5:00pm to 7:00pm.

Contact them through random social networks. Last.fm and other common sites with a social element have ways of sending their users messages. If you can find the account of the person you're looking for, send them a message and it will most likely show up in their regular inbox.

Send an 'I've worked with you' connect request on LinkedIn. Even if you haven't been a colleague, you've got a chance to explain in the note why you want to talk to them. I have a 'pro' account on LinkedIn which lets me send a limited number of messages to people outside my connections, but a sparing use of this approach is much cheaper!

I have a few more ideas I've found effective:

Blog about people or companies you like. I'll often spend time researching  companies or entrepreneurs I think are really cool so I can learn something, and then share it as a blog post. An awesome side-effect of that is that I often hear back from the people I've written about, that's how some of my best collaborations have come about. As I wrote in Beetlejuice, Beetlejuice, Beetlejuice, just saying someone's name on the internet is often the best way to get in touch.

Comment on their blogs or Twitter streams. I find myself doing this naturally with interesting people I'm following, but it's also a great way to build a relationship and demonstrate a sustained commitment.

The key to all of these is thoughtfulness and sincerity. If you really don't care about what they're doing it will come across and you'll just be wasting time. Be natural, be passionate. Follow up, and show you're listening by referencing previous conversations when you do. Spend more time figuring out how you can help them than how they can help you.

Skynet runs on Windows/MFC

Skynetscreenshot

Me and Liz were re-watching season 2 of the Sarah Connor Chronicles when I spotted some familiar-looking code on John Henry's bootup screen. WM_ACTIVATETOPLEVEL sure looks like a Win32 constant, and googling led me to MSDN documentation revealing it's a private message associated with MFC. It looks like autosysconf is running some C++ code to boot up the AI. The other evidence in the series is ambiguous about what side John Henry is on, but AI code in MFC is clearly evil. Interestingly the most common use of FEP is from The Symbolics Lisp Machine as a front end processor, which would be a much more sensible language.

Interestingly though John Henry (and presumably his brother Skynet) appears to be Windows/x86-based, Terminators are known to use Apple II/6502 processors. The thought of dealing with porting between those two almost makes me feel sorry for our future robot overlords.

Scrape your call history with Selenium

Floorscrapers
Photo by WallyG

There's a lot of interesting data out on the web that's locked up in web pages, with no API access to make it machine-readable. I'm particularly interested in phone records; just like emails, IMs and tweets they form a detailed shadow of your social network. To tackle automatically grabbing my phone call history from the AT&T site I turned to Selenium, originally built as a testing tool but also well-suited to screen-scraping on sites with complex login procedures.

To get started you can install the Selenium IDE in Firefox and record the steps you'd manually take to log in and get to the screen you're interested in. Selenium turns those actions into a script you can manually edit and replay. In my case I needed to add some 'type' commands to enter the phone number and password since those weren't captured. Here's the resulting script, you should be able to run this on your own account to download your call details in a csv file once you've added your own details:

Download Attdownload

What's really handy is that you can use Selenium Remote Control to then re-run that same script from your server, using PHP or other popular languages. It's a bit of a hack because it still requires windowing capabilities so it can run within Firefox and a proxy server process to insert the needed code into external pages, but once it's running it's an incredibly flexible way to deal with constantly changing websites.

Move fast and break stuff

Breakglass
Photo by mpires

I recently talked to someone at a very innovative large web company (under Frie-NDA) who described their official engineering motto as "Move fast and break stuff". I love that philosophy because it ties in to research showing that really successful people get there by trying a lot more approaches than average folks. They fail faster, cheaper and more often than ordinary people.

The key to making that work is that the cost of the total failures must be less than the value of the cumulative successes. This is a hard problem, because the default for most organizations is "managing to avoid blame". Their implicit motto is "Reward success and inaction, punish failure", which ends up making inaction the most appealing course. "Move fast and break stuff" encourages a different mentality, "Reward success and failure, punish inaction".

So how do you get that mindset in your organization? The most important step is to de-stigmatize failure. The web company I mentioned makes it clear to their engineers that they will not be punished if they break the site, even if it costs millions of dollars in lost revenue. I didn't get to dig deeper on that topic but I'd imagine there are some serious post-mortem procedures to understand why things go wrong and build tools to prevent a recurrence, like the Five Whys.

Can you help me shape Mailana?

Sculptor

I've got some important and tricky decisions to make about the future direction of Mailana. To make those choices I need to better understand the problems that people are facing, so I've designed a short 8 question survey. If you are interested in the work I'm doing, it will help me a lot if you're able to take a few minutes to fill it out. It also gives you the chance to sign up for early previews of new features before they're publicly released. Thanks!