Photo by PigDump

I was trying to think of ways to make the browser history hack more useful. One of the limitations is that you can only tell if a user has been to an exact URL. So you can tell if someone’s recently been to the main New York Times page at http://nytimes.com/ but that won’t match if they went directly to http://nytimes.com/somestory.html . You can partially work around this by testing a lot of popular internal links (eg all the stories from the front page) but this is a lot harder.

That got me wondering if there was some common property that all the pages on a site are likely to share, something that leaves a trace I can test for. Most websites have a logo image that’s used on most of their pages, and I realized that if I could tell if an image was cached by the browser, I’d have proof that the user had visited some page there recently. How could I tell if an image was cached? Well, if it is in the cache, it should take a lot less time to create it than if it has to be fetched from the network. I gave this idea a quick test, and found that cached images were indeed created synchronously in Javascript, whereas uncached ones took some time. Rather than doing any complex callbacks, I checked the .complete property of each image immediately after creation, and rather to my surprise, this seemed reliable. Here’s an example of it in action, checking for a few common sites:

You can download the full example from http://funhousepicture.com/imagecachetest/imagecachetest.html, but here’s the heart of the test:

function isImageLoaded(image)
{
    return !((image.naturalHeight == 0 || image.naturalWidth == 0 || image.complete == false));
}

function isImageInCache(url)
{
    var image = new Image();
    image.src = url;
    return isImageLoaded(image);
}

There’s plenty of limitations to this approach. For one thing, the test itself pollutes the cache by loading all the images it’s testing, so you can only reliably run this once. All subsequent reloads will show every tested site as having been visited, until you clear your cache. I think I could fix this using cookies to hold the results after the first time, but I haven’t implemented that yet. You also have to identify a common image across the range of pages you’re testing, and with redesigns that URL is likely to change every few months at least. It’s also highly-dependent on how long an image remains in the cache.

It’s exciting to be able to pull out this sort of history information, it’s a good complement to the link style checking, and brings some of the possibilities of the implicit web a little closer to realization.

It took her several days, but Liz has just finished off her map of the trailheads in the Santa Monica mountains. There’s descriptions for each of the locations, describing the trails they lead to, how much parking there is, nearby campsites, which agency owns the land and if bikes or horses are allowed. This was originally going to be just so she could easily link to the meeting points for trailwork from the SMMTC website, but it’s turned into a great resource for anyone who’s interested in getting out into the mountains.

I’m really proud of what she’s accomplished, and it demonstrates how Google’s map-building application opens the door to anyone building rich maps, in a way that just wasn’t possible before. Maybe this will help a few more people discover the beautiful wilderness we have on our doorstep here in LA.

Photo by Abed Dodokh

The original browser history Javascript ran very slowly in Internet Explorer. When it needed to check thousands of sites, like for the gender test or my tag cloud, it could take several minutes. If it was going to be generally useful, I needed to speed it up a lot. The first thing I did was move the test link creation over to the server side, so there was a prebaked html div containing all the links, rather than building it on the fly. This didn’t make much difference though, so I started poking at the testing code. What I found was that switching from array accessing to go through all the links towards grabbing the next sibling of an element seemed to make a massive difference. I’ve included the function below, and it now only takes a couple of seconds to check thousands of URLS:

function getVisitedSites()
{
    var iframe = document.getElementById(‘linktestframe’);

    var visited = [];

    var isIE = iframe.currentStyle;
    if (isIE)
    {
        currentNode = iframe.firstChild;
        while (currentNode!=null)
        {
            if (currentNode.nodeType==1)
            {               
                var displayValue = currentNode.currentStyle["display"];
                if (displayValue != "none")
                    visited.push(currentNode.innerHTML);
            }
            currentNode = currentNode.nextSibling;            
        }
    }
    else
    {
        var defaultView = document.defaultView;
        var functionGetStyle = defaultView.getComputedStyle;

        currentNode = iframe.firstChild;
        while (currentNode!=null)
        {
            if (currentNode.nodeType==1)
            {       
                var displayValue = functionGetStyle(currentNode,null).getPropertyValue("display");
                if (displayValue != "none")
                    visited.push(currentNode.innerHTML);
            }
            currentNode = currentNode.nextSibling;
        }
    }

    return visited;
}

I’m convinced there’s fun to be had with history analysis, so I’ve created a politically focused Facebook app based on my previous tag cloud. Here’s the results of analyzing your history:

I’ll have some more details on the optimizations I did to make the analysis usable on Internet Explorer tomorrow.

I’m a comparative late-comer to Twitter, but I’ve started to get hooked. One of things that pleasantly surprised me is how useful it can be. You can ask questions, or respond to them, and generally do the flea-picking off each others backs that’s required to keep relationships alive, all through a very zen interface.

As someone who reads the back of cereal packets if there’s nothing else to hand, I try to direct my reading addiction into useful channels, mostly towards sources of startup advice and inspiration over the last few years. This has meant personal blogs like Brad’s, Fred’s, Don’s, or topic-based ones like VentureHacks or AskTheVC. The trouble is blog posts are time-consuming, which means there’s a big barrier to passing on a quick link, so posts only happen occasionally. That’s where Sam Huleatt has stepped in, with a use for Twitter I’d never thought of.

His new startuptweet stream is collecting a massive number of videos, stories and blog posts on things that startups care about, like a Stanford introduction to the VC process or Paul Graham discussing how to motivate great hackers. He’s already posted a large number of high-quality resources in just a few days, and I’m hopeful that the ease of posting will make it possible for him to keep up the pace. Check out the full site, and start following!

Photo by 0range Country Girl

I was doing some more research into other companies doing enterprise document analysis, and the combination of staring at this page from PSS Systems and having just finished Bleak House made me step back and realize what a fundamentally dumb idea retention policies for legal reasons are.

As Dickens describes it:

The one great principle of the English law is to make business for itself.  There is no other principle distinctly, certainly, and consistently maintained through all its narrow turnings.  Viewed by this light it becomes a coherent scheme and not the monstrous maze the laity are apt to think it.  Let them but once clearly perceive that its grand principle is to make business for itself at their expense, and surely they will cease to grumble.

Retention policy is a euphemism for deletion policy. Emails over a certain age are deleted, even from backups, usually after 6 or 12 months. The sole reason for this is so that if you’re sued, you aren’t able to hand over older documents, and there’s no question that you deleted them specifically out of a guilty conscience, it’s just your blanket policy. As one of Dicken’s lawyers says:

Being in the law, I have learnt the habit of not committing myself in writing.

There’s no good technical reason for deleting old emails. You’ve made those backup tapes, it’s actually more work to make sure that old ones are destroyed. You also have to make sure you do keep any messages that relate to currently active lawsuits, which is where PSS Systems comes in by semantically analyzing documents to spot those that might be needed in discovery.

Email is the collective memory of an organization, and removing old emails is deliberate corporate amnesia. It’s needed because so many recent court cases have hinged on ‘incriminating’ memos, and with thousands of messages written every day, it’s almost certain that somebody’s dry sarcasm could be painted as deadly serious in front of a jury.

Why does this matter? You’re losing the history of the company. Unless you have explicitly copied them, all those old conversations and attachments you might need to refer back to one day are gone. It’s like putting a back-hoe through an archaeological site, you can never get that information back. Just like archeology, I’m convinced that there will be new techniques in the future that can pull more information out of that data than we can today. Old email should be an asset, not a liability. Unfortunately as long as the legal climate keeps companies terrified of a losing the litigation lottery, they’ll keep deleting.

Photo by Wahj

Robert posted a comment on my BrainCloud post saying that "its a good little pointless thing thats always fun". That’s a pretty fair description for what it does right now, it’s basically a lava lamp for the internet. So why am I so interested in the technology behind it?

The promise of the implicit web is based on knowing information about your users without requiring them to manually enter it. It seems silly that you have to type in all your friends to Facebook when your email inbox makes it pretty clear who most of them are. If I knew which products you’d bought, or which sites you’d visited, I could figure out which to recommend in the future.

There’s a pretty wide consensus that there’s lots of interesting applications we could write based on data like that. The trouble is security concerns make it almost impossible to gather it unless you’re the owner of a well-used site. Amazon can offer recommendations because they have information on all their customers buying habits. No startup can build that application or anything like it without the data, so there’s a barrier to entry that favors the big incumbents.

One approach to get over the barrier is breaking out of the security sandbox with a browser extension. Medium is taking that route, and offering some interesting new search tools thanks to all the data they can gather. It’s really, really hard to get people to install anything though, which makes it a time-consuming and expensive route to follow.

That’s why my eyes lit up when I saw Mike’s social history hack. For the first time, there’s a way of gathering some implicit data without either being a big site owner or requiring installation. There isn’t a killer app for it yet, but I’m hopeful once we all poke at the technique’s limitations, we can figure out some compelling uses.

Photo by Shavar

Web page scripts have a very strict set of limits on what information they can access, which is a big reason why web services are so successful. Unlike desktop applications, there’s no user anxiety that opening up a URL will install a virus or leak confidential information. Almost nobody uses Firefox extensions because we’ve all been trained that any installation step is deeply scary.

One of the things you can’t see from a script is the user’s browsing history. You might not want the world to know all the sites you’re visiting for lots of reasons, whether they’re risque, or give clues about a private medical condition, or even just the fact you’re job-hunting or dating. Even worse, a malicious script could decode URLs to pull out account numbers or user names that could be very valuable to hackers.

That’s what makes the social history hack so interesting. It’s truly a hack in the old-fashioned sense of the word, a clever use of obscure functionality for an unintended purpose. It uses the fact that previously visited links are displayed with a different color inside a web page, creates a hidden link for each URL you want to check, and then finds out if it’s in the user’s history by checking its color.

So, for the first time you can get some information about the user’s browsing history from within a script. Doesn’t that raise the same concerns about privacy and security I outlined above?

I’m glad to say there’s a lot of mitigating factors. You can’t simply find out everywhere a user has been, instead you can only ask if they’ve visited a specific URL. That rules out fishing for account names or other parameters embedded in the address. It’s also reasonably slow, so you can’t practically search more than a few thousand addresses.

I’m hopeful that the very limited functionality of this hack will reassure browser developers and prevent them from patching it as a security risk. It seems like a good balance between opening up some interesting new services based on implicit attention data, without opening the door too widely to malicious exploits.

Photo by Shoothead

In my last post I talked about the barriers I had to overcome to become comfortable with selling my ideas. As Fred Wilson says, some blog comments deserve more prominence than the posts themselves, and Eric Norlin wrote a cracker on the true value of salesmanship. I recommend checking out the whole thing, but the money quote for me was:

Your job *begins* at "no." if all you hear is "yes," then you’re an
order-taker (not a sales person), and you should go get a job at
McDonalds.

Selling is the heart and soul of being an entrepreneur. I’m starting something that I can’t complete on my own, so I need to persuade a whole parade of people to join me and get it done. Everything I’m doing, from this blog, the technical demos I produce, to the conversations I have with smart people I seek out, is part of selling my ideas. It’s not a one-way street, I’m constantly learning more about what other people need so I can do better. I love this process, I really get a kick out of talking with so many folks who are interested in solving the same problems as me. I walk out of almost every meeting with more energy than when I went in. I often hit resistance, but there’s always a reason behind it, and often that’s the seed that inspires me to find an improved solution.

Eric talks about styles of selling, and I’m definitely more comfortable in a relaxed and unstructured setting where I can really connect with the other person. I recognize that the classic pitch is a necessary part of the funding process, VCs are busy people who can’t afford to sit around and chat with everyone who’d like to see them. They need a time-efficient way of understanding the basics of a proposition, but it still feels like speed-dating to me. I feel a lot happier in the Q&A portion, where I can move away from the superficial sound-bites and go into more depth.

I’m definitely still working on my salesmanship, but as Eric says, it can be immense fun once you relax and enjoy it.

Photo by Ardent

I was pondering why Rick’s post struck such a nerve, and I think it’s because I’m still struggling to be an effective salesman without becoming a douchebag. As an engineer and an Englishman, I’ve got two sets of cultural prejudice against salesmen to overcome. The Jargon File’s entry for Marketroid captures the prevailing engineering view of marketers as clueless slimeballs whose job is to trick customers into buying something. The theory is that products should stand on their own merits, and people should be left alone to make rational decisions about which is best for them. Instead, engineers see technically inferior tools beating out their favorites, and blame the evil power of marketing.

As I got more experienced, I began to realize this wasn’t true. Nobody starts from first principles and logically works out what product meets their requirements, that would just take too much time. Instead, we all rely on mental shortcuts to help us make decisions. We’ll look at what our friends are doing, turn to trusted media for reviews, even just look at the packaging to see which looks most professional. There’s also a lot of fuzzy criteria like trust in a company that factor into a purchase, that just aren’t captured in the technical specifications.

Sales and marketing are all about tapping into those human decision-making processes. You look at what actually drives people to make a purchase, and you try to communicate the right information to sway their decision. It’s the art of persuading, and while it can be used for evil, if it’s done right you’re doing people a favor. They need to make the decision anyway, and you’re providing them with the information so they can make up their mind.

The bright line is that you have to be completely truthful in everything you’re saying during the sales process. You have to sincerely believe that you understand the customer’s needs, and they’ll be happy if they choose your product. What’s tough is that part of that decision will be based on non-verbal cues about your own belief and confidence in what you’re saying. If you’re hesitating and stopping to think a lot during the conversation, you come across as nervous and insincere. I try to conquer that by anticipating every question I can, and really knowing the area I’m talking about, since I’m awful at bluffing anyway.

That’s why Rick’s fake question about OS/2 would stress me out. It’s so random, I would have to shift mental gears and think about what he just said for a few seconds before I’d be able to give any answer, probably blinking and looking confused. Then I’d have to give the confidence-sapping answer of "I don’t know" and move on. I also have a tiny nagging fear that maybe I wouldn’t understand, and I’d think I had heard something about it, and give a completely bogus answer.

You don’t have to be a liar to sell, but trying too hard to sound confident leads you into danger. Really confident people will say "I don’t know", and that can even help signal that you’re sincere in your other answers.