How to brick your Ubuntu EC2 server

Bricks
Photo by Mutasim Billah

sudo mv /usr/bin /usr/bin.latest - Don't do this!

What on earth possessed me to run that command? I had just attempted to upgrade to Cassandra 0.8.7 on a DataStax AMI which started as 0.8.1. That involved manually copying files, so trying to be careful I made a backup of any directories I was touching, including /usr/bin. The upgrade didn't work, so I decided to roll back by swapping out the updated and backed-up directories, and the command above was the first stage.

As far as I can tell, there's no way to do anything useful with the machine after that. Ubuntu requires sudo before you can perform any system related, and that command's broken by that folder change, it can't find /usr/bin/python. I had the default Ubuntu setup where you can't run su or ssh in as root, and running /usr/bin.latest/sudo gave a cryptic 'sudo: must be setuid root' error, possibly because of the dot in the path name? Even worse, the Cassandra data files required permissions to access, so I couldn't copy them off.  

It turned into an interesting puzzle for the Unix folks in my twitter stream, thanks for all the ideas. I'm just glad I have three-way replication, and in the worst case nightly backups of any non-reproducible data. The pain of losing hours that I could be spending on features makes this a memorable lesson though. Now I just have to persuade my replacement Cassandra node to get beyond "Joining" after I add it. At least this is giving me plenty to blog about!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: