Super-simple A/B testing in PHP

Photo by Roadside Pictures

To really learn about what your users want you need to see how they respond to the different alternatives. Running A/B tests is a great way to do this, but even though the concept is simple, I always felt like it would require some complex coding and database setup to implement. I was wrong: inspired by Eric Ries's tips from a recent workshop I've been getting a lot of valuable feedback using just a 32-line PHP module and plain old logging to a file.

To use it yourself, all you need to do is think up a name for your test, and surround your alternatives with an if (should_ab('yourtestname', $userid)). That's it. I've deliberately made it so there's zero configuration, you can just pick an arbitrary test name, to encourage myself to test early and often. It's best if you have a proper user id to supply to the test function, but if you omit it, the client IP address will be used instead.

Now when your users load up a page they should see one version or another based on who they are, but how do you gather the information about which one worked? I'm logging all my user events to a file on the server using my custom_log() function, so whenever a user views a page I want to store what options they viewed it with. To do that, the only other function in the module returns an array containing what A/B choices were made for the current page. With that appending as a JSON string to each log entry, I can run analytics on the user's subsequent behavior, to tell which version of a front page led to the most conversions for example. The only tricky part of this approach is that you need to make sure you're logging the event at the end of the page, after all the choices have been made.

If you want to dive deeper, there's lots of strong frameworks out there for split-testing (I particularly like kissmetrics' approach), but even using something as brain-dead as my 32 line module will be a massive leap forward if you're a non-split-tester like I was.

[Update – Doh! I got the random generator wrong, it only returned true about 30% of the time using the md5 test. I've switched it over to crc32 below and in the file]

Download abtesting.php

// A module to let you do simple A/B split testing.
// By Pete Warden ( ) – freely reusable with no restrictions

// An array to keep track of the choices that have been made, so we can log them
$g_ab_choices = array();

function should_ab($testname, $userid=null) {
    // If no user identifier is supplied, fall back to the client IP address
    if (empty($userid))
        $userid = $_SERVER['REMOTE_ADDR'];
    global $g_ab_choices;
    if (isset($g_ab_choices[$testname]))
        return $g_ab_choices[$testname];
    $key = $testname.$userid;
    $keycrc = crc32($key);
    $result = (($keycrc&1)==1);
    $g_ab_choices[$testname] = $result;
    return $result;

function get_ab_choices()
    global $g_ab_choices;
    return $g_ab_choices;

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: