Get visual bug reports with SnapABug

Ladybug

Photo by Hamed Saber

One of the most frustrating parts of trying to fix a customer's problem is trying to understand what on earth the problem is. I've spent enormous amounts of time bouncing emails back and forth, or talking on the phone, just to get enough information to start debugging. I've long been a fan of tools like CrossLoop that let you share screens with a remote user, but I'm really excited to see what my fellow Techstars alumni Timzon have come up with.

SnapABug is a small widget you can include in any web page, and it gives users a button they can press to take a screenshot and email it off to your support team along with some notes. Dead simple but incredibly useful! Jerome, Jerome and Tony have done an awesome job of identifying a great market for their technology, I can definitely see this appearing on a lot of sites and becoming a valuable product.

A coming privacy freakout?

Freakout

Photo by Thatha

People don't know how much information about them is freely available on the internet. I was reminded of that by this thread on the WebFinger list about a prototype YQL implementation that lets you look up information about any Yahoo user from their email address. I'll quote Kaliya:

I went and tried the page out to see what it exposed about me.

Both for my "public" use around the web a lot yahoo handle and another one
that I have explicitly kept my "real name" not attached to in any public
forums.

My name listed in both accounts was Kaliya however when you expose people's
"profile names" in web finger you might be exposing information people don't
think is public on the web.  Needless to say I went in and immediately
changed my profile name in my more private account.

I just shared this with guy friend who has several yahoo accounts – one of
them for dating.  I said do you have your "regular name" listed in the
"profile name" – he thought he might. It sort of made him cringe that this
was now exposed.

I think you might have a real uproar from users by exposing their profile
names publicly on the web without letting them know you are doing this. It
would be good to send people a note asking telling them this information
will be exposed to ANYONE WHO ASKS before you make it available via
webfinger.

I was thinking about the difference between twitter and almost everyone
else.  Twitter starts at Radically Open and explicitly so – so as a user I
know what bargain I am striking in using the tool.

Everyone else is trying to go from "closed" as the default and move towards
more open and pulling users along is a challenge – it is changing the rules
of the space and it needs to be well thought out or it will back fire badly.

The response from the developers has been 'Silly user with your expectations of privacy! Didn't you know there's been a Yahoo profile page with that information up for years?' That's my instinct too, there's so many wonderful new services we can build with more open profile information, and we'll never get anything done if we spend all our time sitting around worrying about potential problems.

From talking to people outside the bubble though, I do wonder if there will be trouble ahead. They don't know that Facebook puts up a public profile for them by default, with pictures, your location and some of your friends all open to the world. Services like Flickr, Amazon, Google, AIM, Vimeo all provide APIs to look up information about someone based on their email address. Rapleaf claims to have social network infomation on 375 million people. If you're interested in the information that's available on you, try this example:
http://web.mailana.com/labs/findbyemail/

The worst case is that we plow ahead with what's technologically possible, trigger a moral panic and we end up with restrictive legislation, but even a mild backlash would cause providers to neuter their APIs and remove access to all that lovely data. So what's the answer? I'd love to see something like the apparently defunct attentiontrust.org pledge to help us self-police our use of the data, before someone else comes in to police us!

The case against the Founder’s Visa

Immigrantbowling
Photo by Vaguely Artistic

In my last post, I talked about why I'm in favor of tweaking the immigration system to provide a Founder's Visa. It's reasonable to ask what the downsides might be, so I thought it might be useful to lay out some possible objections and how they can be managed.

Gaming the system. Will people be able to pretend to be founders to get into the country? As Pat put it, can I just open a restaurant to claim the visa? The safeguard here is money. The proposal is that you need to have over 10% of a company with $250,000 raised in a recent investment. Once the founder's in the US, that investment will flow into the rest of the economy. It's already possible to get a visa if you self-fund a company with a large investment, the main change here is to open the door to raising money from external investors.

So could you figure out a shady deal to fake the investment? Certainly people have been caught using undeclared loans to raise the money needed for the standard E2 visa, but this proposal doesn't make that kind of fraud any easier.

More spaghetti code
. I grew to loathe the forest of obscure rules that make up the immigration system. It's truly mind-blowing how hard it is to get a simple answer to any question, there's so many special cases built-in. It's like a piece of code where years of bugs have been patched by adding yet another if statement to a monster function. The system fundamentally doesn't work, whether your goals are to reduce immigration (it has all sorts of obscure loopholes) or provide a fair way to encourage productive immigrants (it's so complex there's no guarantees you'll be able to stay).

I'm a big fan of the Canadian points system, where they give points for attributes they want in immigrants, like college degree, or work experience in an in-demand industry, and accept those with a score above a threshold. The Founder's Visa is yet another patch to the system, adding a little more complexity to the process, rather than the true overhaul it needs.

That said, I don't see any practical way to do a proper fix of the system, so patching it for this special case seems like the least-worst option.

Giving too much power to investors. There's a delicate balance of power between investors and founders, and knowing that the founder's very presence in the country depends on that term-sheet will give investors a lot of leverage during negotiations. A lot of founders are pretty nervous about this aspect, but I'm more sanguine. If you're a first-time founder getting investment, you're likely to be in a weak position anyway, and are to a large extent relying on the investors being sensible in their demands so that they don't weaken your motivation to make the company a success. In theory this tips the scales against you even more, but in practice the investors have their thumb on the tray anyway.

I'd like to finish by highlighting a comment from someone who ended up in prison for two days after he reduced his class schedule while here on an educational visa! My story's actually pretty tame, I know plenty of people who ended up having to leave the country, so I'm grateful mine has a happy ending.

My immigration story

Starsandstripes
Photo by Kalwa

I'm a big supporter of the Founder's Visa idea, but to anyone who hasn't been through the US immigration process it may be unclear why it's so important. I loved the way Manu Kumar talked about his experiences, so to contribute to the debate I'm going to give a brief run-down of my 7 year journey through the bureaucracy. A lot of friends and colleagues have been through the process too, so I know my story isn't particularly remarkable, but it may shed some light on why so many talented people give up on moving to the US. It's actually pretty long and boring, but that's kind of the point!

When I was 18, I maxed out my credit cards and took a three-month vacation in Juneau, Alaska, teaching archery at a boy scout camp in return for lodging. I fell in love with the US, there was a freedom here I'd never even imagined at home.

Back in the UK I completed a BS in Computer Science at Manchester University, which I'd chosen because of their awesome history in computers, with teachers from Turing to Steve Furber, designer of the ARM chip. After that I spent five years diving into game programming, and became a specialist in programming console GPUs, starting with the original Playstation, and moving onto the PS2, XBox and Gamecube.

The lure of the US was always in the back on my mind, and once I'd finally paid off my college loans and credit cards, I decided to take up one of the recruiters who kept contacting me, and spend some time working for an American company.

I ended up getting a job at Left Field, based just outside of LA. I accepted their offer in March 2001, but I had to wait until August for the H1B visa application to make it through the INS, not for any errors in the paperwork, that was just how long it took them to look at it.

By the time I got there, only 3 months were left on the project, but I had enough experience I was able to dive in and make a contribution. When the project was over, my manager took me aside and told me he and several other folks were leaving to start their own company. I eagerly followed, and for once the visa transfer process was trivial.

I stuck it out at the new company Kush for a year until the first project was done, but at the point where I was working 90 hour weeks and then got reprimanded for being late on a Sunday, I decided I needed a change. Over the last couple of years I'd developed some open-source image processing plugins to help out with my hobby of providing visuals for clubs and concerts, and I'd recently ported my 40 filters over to After Effects. I was astounded by the response, I'd stumbled into a market where people were charging several thousand dollars for a handful of effects, and I could produce popular ones very easily!

So, I left Kush and set out to build my own company with commercial versions of my effects. Unfortunately I quickly discovered that even with my modest savings and a proven market there was no way for me to set up a company in the US. As luck would have it, Apple approached me on the basis of my open-source technology, offered to buy what I'd created so far and give me a job. The most important part was they not only offered an H1B, they would also sponsor me for a green card.

Predictably the H1B process took longer than expected, and I was stuck back in the UK for 6 months again before I could start at Apple. Once I was there, I had to wait a year before the company would start the sponsoring process. Then, I began the first step, 'labor certification', basically proving my job was so specialized I wasn't shoving an American out. Once all the paperwork was in, I waited and waited, and it took over two years for them to finally look at my application. It was approved, and I was overjoyed, until I dug deeper through the results and realized my lawyer had filed me for an EB3 not an EB2. She'd mistakenly put me down as an unskilled worker, which would mean a decades-long wait! I had to refile under the right designation.

This time it went a lot more quickly, and I got the right certification back, and moved onto the next stage in my green card application. After lots more paperwork, including a medical examination to ensure I wasn't a psychopath or sexual deviant, I was back in another queue. This one took around 18 months, mostly because I got flagged for an FBI background check. I understand the need for checks, but it seemed slightly crazy that they had such a long delay, everyone was already resident in the US so anybody dangerous had a long time to get up to mischief.

Finally my green card came through in May 2008, five years after I started the process with Apple. The biggest sensation was relief, I could finally make plans knowing I'd be here permanently. I tidied up my work with Apple, and was free to start Mailana two months later in July.

So what's the point of this story? I love Apple dearly, I can't imagine a better large company to work for, but I'm a startup guy at heart and I ended up taking the corporate path for a long time purely because of visa issues. The Founder's Visa would have offered me a chance to create my own company much earlier, and hopefully employ a lot of Americans too! Beyond that, there was a constant sword of Damocles hanging over me all the way until I got my green card, at any point a paperwork SNAFU or employer decision could have kicked me out of the country.

How to follow your Apache error logs in a browser

Pawprinttrail
Photo by Hunting Glee

As I work to make the data import process more reliable, one of the patterns that was recommended to me was having your import processes append their results to a log file, and then have a database load process that watches the file and updates the database as it spots new content.

This potentially solves a lot of problems for me, but it's not at all obvious how to implement the file tailing functionality in PHP, so I implemented a standalone example to test some approaches. Since I've always wished there was a better way than ssh-ing into my servers to view a live version of the Apache error logs, I made the example a long-running PHP process that outputs updates from the logs to your browser. You can see it running here:

http://web.mailana.com/labs/logviewer/

The code is downloadable as logviewer.zip, or it's included below. Be warned, before you use this on a production server make sure it's password-protected, since sensitive information like passwords or credit-card numbers might leak into your error messages inadvertently!

If you're looking for a Perl version, you might want to look at the File::Tail module.

<?php

//! This function never exits!
//! It sits on the specified file, calling the process function for each new line of
//! input as it's appended by some other process. The cookie value lets you pass in
//! an opaque object to be used by the process function callback.
function tail_and_process_file($filename, $processfunction, $cookie=null)
{   
    $retrycount = 0;
    while ($retrycount<5)
    {
        if (file_exists($filename))
            $retrycount = 0;
   
        $filehandle = fopen($filename, 'r');
        if (!$filehandle && file_exists($filename))
            die("tail_and_process_file($filename, $processfunction, $cookie): The input file exists but I couldn't access it");
       
        $lastaccesstime = 0;
        $lastfilesize = 0;
        $lastfileposition = 0;
       
        while (file_exists($filename))
        {
            $currentaccesstime = fileatime($filename);
            if ($currentaccesstime!=$lastaccesstime)
            {
                $currentfilesize = filesize($filename);
                if ($currentfilesize<$lastfilesize)
                {
                    $fclose($filehandle);
                    $filehandle = fopen($filename, 'r');
                    $lastfileposition = 0;
                }
                       
                fseek($filehandle, $lastfileposition);
                while (!feof($filehandle))
                {
                    $currentline = fgets($filehandle);
                    if ($currentline!='')
                        $processfunction($currentline, $cookie);
                }
               
                $lastaccesstime = $currentaccesstime;
                $lastfilesize = $currentfilesize;
                $lastfileposition = ftell($filehandle);
            }
           
            // Without this, the results of the fileatime() may be cached
            clearstatcache();
            sleep(1);
        }
       
        // The file no longer exists, so wait a progressively longer interval
        // and try it again. After too many retries, die with an error
        $retrycount += 1;
        sleep($retrycount*15);
    }
   
    die("tail_and_process_file($filename, $processfunction, $cookie): The input file was not present after multiple retries");
}

?>

When Sears was a startup

Therainfallsalike

The rain falls alike on the Just and Unjust. All should be supplied with mackintoshes

I'm here in Seattle for a quick family visit (though I couldn't resist visiting a couple of the local startups) and I found myself flicking through a copy of the 1897 Sears catalog left at our B&B. I was astonished at the energy that leapt off every page, these guys were building their own startup! Here's what seemed so familiar:

On a mission with a new business model. They can't stop talking about how they're cutting out the middle men who've been gouging their customers, with pages devoted to messianic rants against the monopolies trying to put them out of business. They contrast their order fulfillment process (dozens of clerks dealing with tens of thousands of orders a day) with the inefficient country stores full of assistants being paid to idly wait for customers, explaining how they can offer such low prices despite the shipping.

The customers are their evangelists. Want to save on shipping? Here's some examples of how you can get $10 of goods for $6 by persuading your neighbors to order along with you.

Information wants to be free
. Want to know more than you ever believed possible about all the differences in pocket-watch mechanisms? Here's several pages of detail that your local jeweler will never tell you, but we want you to understand what you're buying so you'll feel comfortable buying sight-unseen.

Trust in technology
. The very notion of sending money to some company a thousand miles away and hoping they'll send you decent goods in return was a leap of faith even bigger than typing your credit card into a web site. Instead of SSL certificates, they have an engraving of their building and letters from their bankers.

Searsfarm

Selling a dream. They knew people weren't just looking to buy something when they picked up the catalog, so they offer a hope of a better life in their descriptions and illustrations. People didn't just want barbed wire, they wanted that perfect farm, and Sears used that to sell.

They were completely nuts:

Searsdogpower

Like most innovators, they weren't afraid to screw up. Happily I'm pretty certain this dog-powered butter-churner never hit the mainstream, but they have thousands of new product ideas they were constantly trying out, along with really zany ads.

Grasssuits

Sure, online ad placement can be weird, but a real human being decided people interested in grass suits for hunting wild geese will also like a nice pram!

The BOSS, Bing and Google search APIs from a legal perspective

JudgewigsPhoto by Steve Punter

One of the most promising features of the cloud is the ability to leverage other companies' APIs to power your business. More than just saving money, it lets you do things that would be impossible for a startup to build in-house, like a search index for the whole web.

Of course there's always a downside. As Todd Vernon points out in his latest blog post, you're trusting a third-party with your company's future. This is an especial problem with Google since they tend to automate everything, so it can be near impossible to reach a real human being to fix any problems if they do decide to cut off your access. Jud Valeski spotted a classic example of this when some API providers 86-ed IP addresses on App Engine and EC2.

With those fears in mind, it was great to read Jay Parkhill's analysis of the BOSS, Bing and Google Search APIs. There's still plenty of ambiguity left in the agreements, and I've no doubt that the providers could arbitrarily cut me off despite anything in the terms, but it's a great insight into what the providers care about. It should make it a lot easier to skirt any uses that would hit hot-button issues for them, so I'm very grateful to Jay for taking the time to research this.