I’m trying to lay down some decent foundations for processing images for a few thousand users from a single server. Because of the thrashing problem, one of the building blocks has to be a single queue for the CPU intensive image processing commands. Thrashing is when you run out of physical RAM, and you end up spending the majority of your time swapping virtual memory blocks on and off disk. This usually happens when there’s multiple memory-hungry processes competing, and as they’re time-sliced, they repeatedly push each other’s memory out of RAM, and pull in their own back. The result is that the actually take a lot longer to all finish than if they’d each run in sequence.

Well, this is an old, old problem, so I was hopeful that there’d be a tried and trusted unix mechanism to deal with it. The simplest way to avoid thrashing for my case is to make sure such operations do run one at a time, by keeping a single queue, and only running the next in the queue after the previous one is complete.

After some digging, I did discover batch, and it looked very promising. I ran into a few wrinkles. It expected a script file on disk as an input, rather than the dynamic commands I’d be feeding it, but I was able to avoid that by piping the output of an echo "<command to execute>" into the batch command. More serious was error reporting; it will sendmail the results of batch executions, but there doesn’t seem to be any other way of accessing that data. My simple tests worked, judging by the results I saw on disk, but some of the more complex ones failed, and I haven’t been able to work out why yet, because none of my server accounts have a mail set up.

Once I’ve got that working, my plan is to put all image processing commands into the queue, rather than executing them synchronously with the PHP page generation as I am at the moment. Then I’ll need to have a timeout call in the HTML itself, to refresh the page frequently until all the images have been completed. With all that in place, I should be able to show users some evidence that things are happening even when the server’s under a heavy load, rather than the blank page they see at the moment.

Funhouse Photo User Count: 268, active 78. Good to see it still climbing, and I’m not losing too many people. The active count is down, but that’s still around 40% of the users at the time the count was taken, so a good base to build on.

Facebook has just introduced some new default statistics for applications. Previously they would just show the total number of users that had installed the application, now they’re trying to measure how many people are actually using the app every day. This is an interesting change, since as they say you start off trying to measure what you value, and end up valuing what you measure. For Facebook apps, measuring the total number of users favored those that were shallow, but spread widely, whereas this new measure is designed to capture how much users actually engage with an app.

The measure is shown as a percentage of your users who actually take an action that interacts with your app every day. There are various actions captured, but they all involve the user actually doing something with your app.

The first day’s figures were very kind to Funhouse Photo, showing 86% of the users as active the previous day. I’d expect this to drop a bit as some of the novelty wears off, but I’m planning on regularly adding new effects in the hope this will keep people playing around with it, and sending photos to their friends. I’m also thinking about showing every photo change in your feed; I was worried that was a bit spammy, but I’d like to experiment to see if it is actually useful.

Funhouse Photo User Count: 200. Disappointing to be honest. It’s still climbing, but with a fairly shallow slope. I shall see be able to see if new effects and some more visible notifications give it a boost.

I’ve now got the new server up and running at http://funhousepicture.com/. I’m holding off on moving the app from dreamhost until the DNS has had a good chance to propagate, but I’ve got a test version running successfully.

It was a really interesting experience getting the server running. cari.net is definitely set up for people wanting to resell server space, rather than my simpler needs. They also assume a good knowledge of linux server administration, which is a reasonable expectation, but one which I struggled to meet! Luckily google came to my rescue and I was able to struggle through

• Setting up ssh for multiple users
• Installing and building ImageMagick and its dependencies
• Removing the virtual host bumpf that Plesk had set up for me
• Sorting out the file permissions so the apache process could actually work with my non-root user
• Removing auto_index from apache, since I really don’t want directory listings exposed to the world by default!

I was able to get that all done early this morning, and this evening, so the move should be possible tomorrow.

Funhouse Photo User Count: 164. It’s gone up more slowly than yesterday, but I am seeing plenty of people coming in from notifications their friends have sent them, which is promising.

Funhouse Photo appeared on Facebook’s directory a few hours ago, and it’s up to 106 users already. The bad news is that I’m starting to see occasional timeout messages when I’m testing it. Looking at the CPU usage on my server using the unix command ‘top’, I see spikes that seem to be caused by the heavy image processing demands of creating the photos. I’m using a shared server on dreamhost and I was hoping to avoid upgrading until I had more users. Unfortunately it looks like this is the point where I need to switch to keep things reliable.

The next step up is to get a dedicated machine, rather than sharing the CPU with other users. I’ve been very happy with dreamhost, but their dedicated server packages start at $400 a month. That’s a bit much, so I’ve gone with a mid-range package from cari.net instead, at $135 setup and $135 monthly.

I will be very interested to see how this works now, and scales as usage increases. I may need to rewrite the image generation to make it more asynchronous, so that the CPU load is smoothed out rather than spiky. One way to do this would be to only create a few images at a time, and progressively load each page, but I’ll think on that some more…

Funhouse Photo User Count: 106, after today’s inclusion in the directory.

Here’s some techniques I picked up during Funhouse Photo’s development.

Create a dummy user for testing notifications

Unless you’ve got a trusting friend willing to give you access to their account, you’ll need a second account for yourself to make sure that your notification and request code is working. This made me a bit uncomfortable, since one of the things I like about Facebook is how closely accounts correspond with real people. I did try to get by without using a second account for a while, and gave up when I realized I was probably starting to spam my friends, since it was sometimes unclear if a message had actually been sent.

Have a second copy of your app to use as a sandbox

I’m sure this is standard web app development practice, but as a client programmer, it wasn’t immediately obvious. Create a second app in Facebook (Funhouse Photo Testbed in my case) and use that for all your development work. When you’ve got a new version stable, change the config and copy it over the old version, so users see an instant change from one version to another.

Enable debug output from the API

Add this line to the top of your PHP files, before you include facebook.php

$GLOBALS['facebook_config']['debug']=true;

You’ll now see output for every facebook call you make. I wasn’t able to twirl open the categories, so I also edited call_method() in facebookapi_php5_restlib.php to have their styles default to visible.

Use GET rather than POST for passing data

I find it a lot easier to check the address bar to understand what data’s being passed than find ways to dig into the POST data (eg using LiveHttpHeaders). This is against the standard meaning of GET, since it’s not supposed to have any side-effects, but I’ve left it active in the released version anyway, since I enjoy life on the edge. It’s probably preferable to switch to POST for releases, and I’ll be looking at that in the future.

I had a really low moment yesterday. I’d just finished the request-sending part of Funhouse Photo, and it was working like a dream. The request contained a picture of the recipient that the sender had put through Funhouse, along with a short message. I sent a few test messages, and everything looked good.

Then, it stopped working. Instead of the app-supplied image, the request now showed the plain photo of the sender. This completely killed the effectiveness of the request! Before it was really compelling because you actually saw a photo of yourself run through the software right there in the message, now it was unclear what Funhouse Photo actually offered.

I discovered that Facebook had decided to start ignoring the app-supplied image. They didn’t explain why in their announcement, but I can only guess there were some security concerns, though I’m having a hard time picturing the problems. This is why I hate, hate, hate having platform dependencies, especially on something that is so new and fluid. They can make a decision that has a devastating impact on your product, and you’ve got no way out but trying to work around it.

Anyway, I sucked it up, and completed an alternative implementation that uses notifications instead of app requests. There’s still no photo visible until you click, so it’s definitely not as good as my original setup, but should still provide a fun service for my users.

Talking of users, I’m now up to a grand total of 7! The app is still waiting in the queue for the directory, but the interesting thing is that two of my users aren’t in my friends network, which means they must have spotted it on one of my friends’ profiles and decided to add it. That’s very encouraging.