Photo by Ulterior Epicure

I'll admit it, I had written off Google's latest effort to be social. I have a tidy little theory that their focus on metrics leads them to local maximums but prevents them leaping across gaps to islands of fun. I expected Google Plus to be a flash in the pan as people signed up, poked around and never returned. That left me surprised to hear a continuing murmur of interest from people I trust like Marshall Kirkpatrick. I almost hate to publicize his secret, but he has managed to notch up a series of great stories by using the service for research.

On Thursday night I'd just finished off a links post and wanted to do my normal tweet about it, but Twitter was down. When it was still unavailable after an hour, I decided this was a sign I should check out Google Plus myself. I went to the site and put together a short post with an explanation I was experimenting with the service.

Speed

My first impression was of how fast the interface was. It didn't have that lag that I thought was inevitable with web applications. It was so quick I was convinced they must have pre-populated the page with hidden content so it was instantly available. When I looked at the network activity it turns out I was wrong, most of their content is dynamically loaded by Ajax calls. What makes the difference is how fast their response times are, often under 100ms for me. I'm definitely a snob about this sort of thing after my time at Apple, but the snappiness and some thoughtful work on the interaction model made a profound difference to my experience. The fit and finish make me want to spend time exploring the service, in contrast to Twitter's web interface which just feels a lot klunkier and messier.

Length

Being able to write at a natural length was a surprising relief. I still stayed fairly short, within the same three-sentence rule that I try to use for emails, but I was over 140 characters. I don't think the service would work unless we'd already been taught brevity by Twitter, but being able to expand beyond those limits when I need to feels very liberating.

Audience

There were a surprising number of responses to my initial post. One of them was from DeWitt Clinton, in what I wrongly suspected was an auto-generated welcome message. I knew him through the Buzz and WebFinger developer mailing lists, so I thought he might be the Tom Anderson of Google Plus, the first friend for every new user. He convinced me he was human, so either the employees are very engaged with the service, or Google's code can now pass the Turing Test. It seems like there's quite a few regular people watching the streams too, which is confidence-inspiring this long after the launch.

Conversations about cheese

The comment thread grew into a couple of conversations, giving me a chance to reconnect with several friends. In particular Audrey Watters, Edd Dumbill and I started waxing nostalgic about British food, and the struggle to find decent cheese. Edd was inspired to create a short public post, and the whole experience convinced me that the service makes some great conversations possible. It left me wanting to return regularly, which gives me hope the service is sustainable.

I'm happy that I was proven wrong about Google Plus. I won't be abandoning Twitter, but I will be spending more time on Google's service because it offers a different experience, one that I was surprised to find myself enjoying. If you're curious, join me there too.

Photo by Holeymoon

Sourcetree – I don't often recommend commercial software, mostly because my personal stack's mostly open source these days. I've fallen in love with this tool for exploring my git repositories though. Git's new Mac app is fantastic too, but focused on 'doing things'. I've found sourcetree a wonderful way to explore and understand your code. I just discovered they've been acquired by Atlassian, so I guess I'm not the only fan!

The Sketchbook Project – I never lose my sense of wonder at how many ways the web can be used to drive creativity. By offering to scan people's sketchbooks they've motivated a community of artists from all over the world, and given me a vast set of material to browse through when my imagination needs a jump-start.

Smart Meter surveillance – How your electricity meter can reveal what TV channel you're watching. My German's not good enough to follow the main paper, but the abstract sounds very plausible. From my perspective not something to freak out over, but a good example of all the unexpected ways we leak information about our lives. We measure more and more things to improve efficiency, but the by-product is that the same data can be used for many unintended purposes too.

The Brown Revolution – An unfortunate name, but a compelling idea for sustainable grazing. I'm normally skeptical of agricultural 'silver bullets' like this, but I know from my experience maintaining trails how effective thoughtful drainage can be. When water's compressed into a narrow stream by a gully it will cut through even packed soil like a plasma torch, but keep spread out it in a wide sheet using a shallow 'rolling dip' and you'll have a surface that can survive years of storms.

Apple insiders remember Steve Jobs – I'm very sad we've lost Steve, he always seemed more like a super hero than a mortal to me. At the Guardian's request I contributed a few thoughts about my time working at Apple, and how he was a constant presence even though I barely met him. I'll be thinking of his family.

Photo by Nick P

Is teaching MapReduce healthy? – The working conditions inside Hadoop are terrible, but it's rapidly becoming the default for large-scale data processing. Does that mean students should learn the MapReduce approach? It feels a lot like the debates over teaching ugly, confusing, widely-used C/C++ versus beautiful, elegant but niche functional languages, and which should come first in the curriculum. This article gave me some fantastic glimpses into the wider world of distributed frameworks and techniques, and left me itching to try Bloom.

Amazon comments on Spot price spikes – A refreshingly detailed and open response from a large company. I'm disappointed that the prices have suddenly become so volatile though, since that severely limits the places I can use them.

PhantomJS – A WebKit based headless browser, lighter-weight than Selenium and driven by Javascript. I hope I get a chance to use this, and that they can sever the vestigial dependency on X windows soon. My main use case would be generating screenshots.

Teaching data to speak humanely – Looking at the Facebook timeline, and how old metaphors of interaction disappear as the interface gets closer to the content.

Pictures of the Big Bang – Gorgeous snapshots of our universe's very first moments, courtesy of a computer simulation:

Photo by Paul Hohmann

If you’ve been a user of my Data Science Toolkit or OpenHeatMap sites, you may have noticed they’ve been a bit flakey recently. The back story is that I’ve started a new company (of which more soon) and I had to cut back on how much I was spending on my own servers. It was costing me over $1,200 a month on all the different systems I’d set up over the last three years! This was mostly because it was quicker and easier to set up a new instance than worry about jamming it onto an existing one, and I never got around to cleaning things up. I doubt there are many people who’ve been as lazy about this as me, but if you’re looking at cutting costs, I would start by figuring out if there’s servers you can merge.

By cutting out several like twitter.mailana.com and fanpageanalytics.com that were no longer being updated or heavily used, I was able to cut that in half, but I then reached two sites that have decent traffic. I started by merging DSTK and OHM onto one large server, which mostly worked but caused some hiccups. That got me down to around $400 a month, including another small instance for some legacy Mailana labs sites.

I then decided to switch to ‘spot instances’, Amazon’s auction model for buying spare server capacity cheaply. Most of the time it’s only about 12 cents an hour, a third of the normal price. I switched, but then started to experience some serious price spikes that kept taking the server down, and required manual intervention to set everything up again. At some points, the price went to $15 an hour, so there’s obviously capacity limits being hit. That’s very different from my experiences just a few months ago when prices seemed a lot more stable. At this point I’d never recommend spot instances for user-facing servers, the downtime seems too high. They’re still a great deal for things like MapReduce backend processing.

I need 64 bit support for a lot of the frameworks DSTK relies on, so I couldn’t go down to a small instance, but I did realize that micro instances were x86_64, so my next step was to try running both sites off one of those. Rather predictably, the processing requirements and lack of memory crippled the tiny instance, so the site was extremely flakey. It would have been only $14 a month though, so I spent some time trying to fix the issues, by configuring swap space for instance. My conclusion was that micros are too limited for anything but light web serving.

Today I finally bit the bullet and bought a one year reserved m1.large instance, which cost me about $900 up front and another $1,100 over the next year in costs. I also rolled in my web.mailana.com small instance into the same server, so I’m down to about $170 a month! This is around a $100 less than the unreserved cost, so I’d seriously look at reserving servers as a way of managing your costs, if you can stomach the up-front deposit.

I’ve also added a link to my O’Reilly books to the sites, in the hope I’ll cover some of my costs:

Improve your data skills (and keep this server running!) by buying my guides:

Data Source Handbook

Big Data Glossary

Photo by Leo Reynolds

Fingerprinting Cameras through Sensor Noise – Arvind shows how unique the noise pattern that your camera embeds in every photo you take really is. It's as identifiable as a thumb-print, and with high resolution pictures available from many sources associated with your real name, makes it theoretically possible to match up photos you've tried to take anonymously. The one saving grace is that compression removes a lot of that unseen uniqueness, but it looks like you'd need to go down to 60 quality on JPEG to be sure.

Multiplicative Microdata Noise for Privacy - Noise isn't always bad though. I'm fascinated by the ideas of adding it to prevent reverse-engineering of sensitive information from data releases. I bet it's tough to get right though, I'd love to set Arvind or some of the Kaggle team loose to see how solid the protection really is.

Underscore – The hardest thing about switching to Javascript from other languages is its primitive support for iteration. I'm excited by the promise of this library that Tyler Gillies pointed out to me, it looks like a great solution for my grumbles, unless there's a catch I'm missing.

requirejs – He also showed me this solution for proper module loading in JS. The syntax is a bit funky, and I know we'll still want to compile and minify for production, but it's still a good tool to have in your kit.

TeleHash – An intriguingly generic way of connecting machines together in a peer-to-peer way using distributed hash tables. Maybe it's just my exposure to them as I do more Cassandra work, but DHTs seem to be popping up all over the place.

Photo by Ronaldo F Cabuhat

CryptDB – A PostGres modification that stores data in a securely encrypted way, but still allows efficient queries. I love the idea, can't wait to see the software.

Perceived density – We very rarely go back and look at what the numbers we're using actually mean. This author noticed that the standard population density figures for cities didn't match up with anyone's real experiences with those places. He then reasoned about why that was, created a different way of measuring and built figures that seem a lot closer to reality. New York really is denser than Los Angeles!

Reverse Social Engineering (pdf) - If attackers can persuade their targets to approach them, the attack is more likely to succeed. The friend recommendation systems of social networks like Facebook can be gamed to manipulate those targets into friending attackers. Via Prasanna

If This Then That – Wonderfully simple scripting system for joining together web services like Flickr, Facebook and Twitter.

Hector – A battle-tested Java interface to Cassandra with a strong community behind it.

I'm running into consistent problems when storing values larger than 15MB into Cassandra, and I was hoping for some help on tracking down what's going wrong. I've emailed the Cassandra list, but I thought my long-suffering readers might like a crack too. I promise I'll get another five short links post up there if I can crack this bug!

From the FAQ it seems like what I'm trying to do is possible, so I assume I'm messing something up with my configuration. I have a minimal set of code to reproduce the issue below, which I've run on the DataStax 0.8.1 AMI I'm using in production (ami-9996c4dc)

Photo by Beverly and Pack

Ten years ago, I stepped off a plane at LAX and began a new life in America. I was definitely English at that point. A few months back, an NPR interviewer confronted me with a question – "Should we describe you as a British or American researcher?". I'd never had to answer that before, so it took me a moment to think, but only one answer felt right – "I'm American". How did I know?

Being Good

The first change came in the way I talk. I'm terrible at accents, so I still sound funny, but my phrasing changed radically. It was a natural process, people only expect to hear certain responses and so I ended up being trained to avoid puzzled looks. I knew I'd crossed a threshold when my parents started mocking me. "How are you Peter?" – "I'm good" – "We know you are! Teheheheehehe". The correct British response is "I'm well", "Ok", or a grunt/mumble.

Enjoying Fake Pubs

I still miss mum's roast dinners, Indian restaurants and chip butties, but the stereotype of terrible British food is generally true. We're known for our drinking though, so most cities have a few theme pubs. There's apparently a legal requirement that they feature a traditional phone booth outside (though usually without the improvised restroom quality that would really mark them as authentic). When I first arrived I hated these places. I had a horror of being an 'ex-pat' who sat around watching soccer and reading The Sun, never truly engaging with the culture of the place I lived. They always seemed like shabby emotional props, over the top stage sets with enormous portraits of Churchill and the Queen, so many clichés it felt like walking into an Austin Powers movie.

After a few years though, something happened. The terrible taste no longer seemed important, I loosened up and began to look forward to my occasional visits to American pubs. I had developed a new sense – the ability to detect and appreciate quaintness. Growing up in the English countryside I was surrounded by so much of it that I had no awareness, in the same way that I guess fish don't think about the water they swim in. My normal was now strip malls, sushi and bars, so anything British felt foreign and interesting, and the authenticity no longer felt so important.

A Fascination with the Old Country

As that change progressed, I realized my reading habits were skewing towards British history and literature. I'd always been a voracious reader with traditional classics in the mix, but I felt a need to fill in the gaps in my knowledge of the British past, both non-fiction and culture. The typical American obsession with ancestry always looked ridiculous from the other side of the Atlantic, but now I understood. It's not a rejection of the US national identity, it's an integral part of it. Because there's people from such a wide range of different backgrounds having to live together, history and ancestry provide a safe way to talk about our differences. It's non-threatening because the stories all end up with us here in America. Talking so much about the variations implicitly acknowledges how much we have in common.

Atrophied Irony and Sarcasm

I inherited an ability to come up with a cutting remark for any occasion from my mum, and I still find her a riot. It served me well in the UK, but I rapidly discovered that it generated confused and concerned looks over here. It dawned on me that sarcasm relies on a deep shared understanding between the speaker and listeners, or it won't be clear that you actually meant the opposite of what you just said. There just isn't enough agreement about what's normal here to deploy sarcasm for anything but the simplest situations. You can find Americans who will believe almost anything, and be willing to say it! I also realized I was hiding behind my negativity. I was afraid of stating what I really believed and desired explicitly for fear of rejection or mockery, so being ironic gave me plausible deniability. I'd still hope that people would agree, but was protected if they didn't.

This timidity had to go. You can't persuade people to help you get interesting things done unless you clearly show you believe in them yourself. My big secret was that I've always been painfully earnest, and moving to the US gave me the chance to come out of that closet. I get to talk about the crazy things I dream of building without immediately hearing a million reasons why they're bound to fail. I still love a sprinkling of snark, but as a little spice, not the main dish. One of the hardest things about going back to Britain is biting my tongue so I don't sound arrogant, because I'm not hedging what I'm saying enough.

The Passport

This is the last piece of the puzzle. I have my green card, and I'm just over a year away from qualifying to become a naturalized citizen. I'm going to apply as soon as I can, even though there's almost no practical difference between my status with a passport and being a permanent resident. America has given me the chance to live some amazing dreams, make wonderful friends, to create things and have experiences that would have been denied to me in the UK. It feels like home. I want to make that official.

I'm a big fan of using EC2 spot instances to help reduce costs, but the pricing behavior can make managing them a pain. There's a very bimodal distribution, where the price will wobble along at around 16 cents an hour almost all the time, with infrequent but sudden spikes up to 50 cents or so, above the non-auction cost. I don't know for sure why things are so erratic, but I can think of a couple of possible causes.

It could be that there's a very non-linear distribution of maximum bids, where almost everybody is willing to pay up to the normal price for an instance, and so when capacity is reached and machines need to be shut down, the price has to shoot up before any significant number of resources are freed up.

There could also be some very heavy hitters who occasionally demand very large numbers of machines, so causing the price to spike. What was interesting this evening was that I was still able to manually start up a couple of instances at the regular price, which was below the auction price. I also find it surprising that the two data centers seem to be so highly correlated. Unless there's sophisticated users who are quickly switching their requests between availability zones, I'd expect a lot more independence as the spare capacity in each varies.

Chart via Felix Salmon

Of course, there could be something deeper at work. Werner Vogels was born in Europe, and the Swiss National Bank is trying to find somewhere to sink all its money to prevent the currency from appreciating. Coincidence? Mind you, I talk a bit funny too, so it's hard to trust anything I say. Unless I'm narrating a documentary about meerkats, in which case the accent helps.

Photo by Keven Law

Photo by Stew Dean

The Good Judgment Project - Almost all of us are terrible at making predictions, even professional pundits, and worse, we're unaware that we're so bad at it. That makes me excited to see this academic project to objectively analyze the techniques and people who make the most accurate forecasts. Even just a minor improvement in our prediction skills could make a world of difference in the quality of our decisions, so I'm looking forward to seeing what results come out of the study.

Thesis on phone geotag analysis – A strong overview from a UK undergraduate, covering a lot of different ways that location can be determined from iPhones and other smart phones.

Identifiability of de-identified data – Summary of a researcher whose work demonstrates how flawed most data anonymization is, even for sensitive medical information.

Shape optimization of gridded surfaces – I had a good conversation with Avik Das, the author of this work recently, and it made me nostalgic for my time in computer graphics. This video shows some of the research he's doing to 'relax' complex geometric shapes into a more natural arrangement, and is beautiful in a very geeky way.

Bulk loading in Cassandra – Yet more useful material from DataStax, on a problem I've been looking into a lot recently. Since I've had no luck getting Mumakil running, I'll see how this approach works.