Cassandra connections are costly

Photo by Sara Lando

I've been working on speeding up our page load times, and one of the slowest sections turns out to be the calls to the Cassandra database. I was surprised to see a simple value fetch taking up to 500ms. When I dug down into the Ruby gem we're using, I discovered that 400ms of that was inside a call named extract_and_validate_params(), and then I traced the time down to column_family_property(). At this point it descends into the world of auto-generated Thrift code, but as far as I could tell from inspection, the time taken is in fetching the column family schema definitions which are held as values in Cassandra's key/value store.

At that point I hopped over to the #cassandra IRC channel, and Ben Black confirmed that my suspicions were correct. The first get on a given Cassandra connection also has to fetch the schemas, so there's at least one round-trip to the database in addition to the operation you're actually performing. Ben's response was that frequent connection destruction and creation was "not a good idea, regardless of database", which is somewhat true, but generally the sort of connection pooling that you need to avoid it is not much fun to handle at the application layer level. When connections are expensive to create it's because they have state associated with them, and in my experience that lingering state leads to all kinds of hard-to-track-down bugs as one piece of code sets something which then has knock-on effects in an unrelated module when it reuses the connection.

Luckily our use of Cassandra in the frontend is fairly straightforward right now, so I'm going to try the simplest possible kind of connection pooling, a single global connection that's opened once for each Ruby process, and then reused for every subsequent call. The only wrinkle I'm anticipating is error handling. If something goes wrong with a call and the connection goes down, I need to detect that and re-open the connection, and it's not clear how to detect that. Anyway, I wanted to share a bit of my work-book with the world as I'm wrestling with this, and I'll be curious to hear if anyone else has had to tackle this problem, especially in Ruby.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: