The case against transparency

Photo by istargazer

Eric just posted on the advantages of transparency. I’m a fanatical believer in the power of more openness to transform businesses and my whole email startup is based on the idea that there’s hidden information in our emails that’s worth revealing. The problem is, within the tech community ‘open’ is a synonym for ‘good’, and that gets my contrarian antenna twitching. As Fred Wilson says, you don’t make money by doing the same thing as everyone else, so here’s a couple of examples of transparency gone wrong.

Misleading metrics

The mortgage industry moved from a centralized business model to one where different stages were handled by separate firms. Landing clients was handled by mortgage brokers, firms like Countrywide would then write the mortgage, but the money itself was provided by investors through securitization. In the old days a single firm would handle all of this in-house, which meant they had deep and immediate access to all the information about a borrower at every stage. To make the decentralized model work, an open and standardized way of categorizing the quality of the loan was developed. Statistics and measures to cover the credit history, income and collateral offered by the borrower were passed up the chain. These were then used by the agencies to rate the loans and split them into tranches of risk. It looked like a model of transparency, increasing the efficiency of a whole industry.

The problem was the metrics were systematically false. Brokers had massive financial incentives to inflate collateral house values through friendly assessors, and help borrowers inflate their income. Unlike the old single-firm approach, there was no real accountability for the true quality of the loans, they would still get their commission. The rating agencies relied on the broker data, and had similar incentives to grade the loans favorably.

Part of the reason we’re in this mess is that the appearance of transparency made everyone complacent. A manager of a team of sales people in the single firm model would get fired if her team were fudging originations. Her management would have a strong incentive to prevent lax lending standards because that would lose the firm money. There just wasn’t an accountability mechanism to go along with the new open model, and so the apparent transparency was a dangerous illusion.

Destroying the magic

Walter Bagehot said about royalty "The monarchy’s mystery is its life. We must not let in daylight upon magic." The same could apply to Apple. One of the distinctive elements of its culture is the obsessive secrecy. This isn’t just the usual bureaucratic urge to hide information, it’s a deliberate part of their marketing strategy. The impact of any announcement is so much larger when it’s a surprise. When nobody knows what Apple’s really working on, people’s imagination runs overtime anticipating what could be coming next. Any projects that went south before release were never known to the public, helping us look far better in comparison to more open companies.

There’s massive downsides to this too, I always struggled with simple things like getting trusted developers onto beta programs because of the secrecy, but it’s hard to argue with the results.

Tarantula Hawk


Apologies to any arachnophobes, but last night I was lucky enough to run across a really gruesome bit of nature I had to share. We often spot Tarantula Hawks flying around, but I’d never seen how they got their name. They’re enormous wasps, several inches long, and the adults live on nectar. They’ve worked out an ingenious business plan for feeding their larvae:

1- First hunt down a wandering Tarantula.
2- Paralyze it with your venom.
3- Dig a hole, shove the spider into it, and lay an egg inside its body.
4- Cover up the hole.
5- The larva hatches, first sucks all the juices from the still-living spider, and then eats it from the inside, saving the vital organs until last so it stays alive and fresh as long as possible.

The photo is from a wasp we came across that had just paralyzed its victim, and was getting ready to drag it to its lair. How cool is that?!


Creating an Outlook plugin with Add-In Express


I’ve always written Outlook plugins from the ground up in C++, since I’m very wary of dependencies on frameworks like .Net and other components that can turn deployment and debugging into a nightmare. I recently ran across the Add-In Express suite of tools for building Office plugins, and it offers enough to change my mind.

I paid $349 for the standard version, and the first pleasant surprise is that there’s no royalties for your end-users. Another big plus is that you can upgrade to a premium version that includes full source code for just $949. This is very important if you’re creating a commercial product, it means if they go out of business you can still keep tweaking the code to deal with OS upgrades or minor bugs. There’s also various discounts available for things like blog reviews, though I didn’t take advantage of that. [Update- After posting Andrei from ADX was kind enough to give me a free upgrade to the Premium edition]

The purchase and download was very painless, and it installed itself as part of Visual Studio, offering new project templates for the various Office plugin types. The license is limited to 3 development machines, so I will have to see how that works with my frequent reinstalls of my Parallels VM. I chose ADX COM Add-In from the extensibility section of the project templates,  and then went through a couple of wizard screens choosing which language and applications I wanted to use. I went for C# (new to me, but since I needed a heavily UI-based plugin C++ was just getting too painful) and Outlook.

You choose to have an installer automatically generated when you create your project, and this is an incredible time-saver. I’ve lost countless hours fiddling with the guts of WIX installer, so this alone could be worth the price. Unfortunately I wasn’t able to get it working, and it looked like it was related to my use of UAC on Vista. I discovered a workaround I could use during development though, the ‘Register ADX’ context menu item on the project worked like a charm. It looks like UAC may be an ongoing problem for Vista deployment, but their forums are extremely active with both developers and support staff, so I feel confident I can get help on issues as they come up, and I’ll be on a well-trodden path unlike my home-brew efforts.

Their claim is that they will handle all the horrible add-in plumbing and let you focus on writing your application code, and so far they live up to that promise. I was able to get a simple window added to Outlook within a day, a lot faster than my previous plug-ins I wrote from scratch. There were plenty of head-scratching moments as I read through the documentation, but nothing that I couldn’t figure out with some experimentation and looking through the forums.

All-in-all, a big thumbs up for Add-In Express. I’ll have a more detailed review once I’ve really used it in anger, but I’m optimistic I’ve found a real time-saver.

Adding cookies to CPeteHttpRequest

Photo by Esti-s

I’ve recently been emailing back and forth with Edward Hibbert as he builds on some of my BHO work to create an Internet Explorer plugin for Freecycle moderators users. I’m a fan of the organization, it’s a real win-win if unwanted items that would otherwise go into a landfill can be passed on to someone who can use them. I was the treasurer of a vaguely similar alternative currency group when I lived in Dundee, but this is a much simpler way of unlocking untapped value in a community.

One of his requirements was that he needed to pass around cookies in his internal HTTP requests, something that my base CPeteHttpRequest class doesn’t support. He’s implemented this functionality and kindly passed me the modified code, which I’ve included below.


[Update- corrected a couple of details like plugin’s homepage]

Run the Backbone Trail


I like to think I’m fit
, but ultra-runners like my friend Howard Cohen make me look like a couch potato. I just discovered a great page he’s got up on running the Backbone Trail, going the length of the Santa Monica Mountains on the western edge of Los Angeles. It’s about 70 miles with some serious hills, and he’s managed to do it all in 16 hours on a single day. That blows my mind considering how tough I find some of the sections just to hike!

It’s through a beautiful expanse of wilderness just a few miles from the city. If you’re one of those crazy ultra-runners and you make it to LA, check it out, you won’t regret it.

How close are we to Vernor Vinge’s future?


Vernor Vinge writes about the future, and the worlds he builds fascinate me. They’re packed with weird but plausible tools, and he understands the world-shaking power of communication. If you want a flavor of his writing, check out his first novella True Names online.

He’s best known for predicting the singularity, but I think his most interesting innovations are much more personal, and I see a lot of Defrag’s community starting to turn them into reality. Here’s how some of is ideas are moving towards the real world.


In Rainbow’s End, a lot of the dialog takes place by ‘Secret Messaging’, or sming. It’s like instant messaging, but with an interface through wearable computers that lets anyone silently send with no sign to the outside world. One of the most revolutionary uses of IM I found at Apple was having hidden side-channels to your colleagues in meetings, so you could frankly compare notes while someone else was talking, and plan your next moves together. This required a laptop open and visible typing though, but Vinge understands how much a truly hidden version of this changes. The plot is driven by people conspiring through sming, everyone’s addicted to it.

You can see how we might end up there with companies like Parlano turning IM into a vital workplace tool, and SMS through mobile devices making it possible to send from anywhere. The real leap will be the interface, and though it seems sending through something like tapping or subtle gestures should be possible, receiving requires retinal displays.

Relevance mining

Even way back in 1979’s True Names, Vinge’s background in computer science let him see how you could find patterns in massive sets of data. One of his recurring themes is that the characters have technology that searches through large sets of messages and pulls out a small group that are potentially interesting. This sort of assistance is a lot more plausible than the common SF trap of tools that require computers to grasp meaning, since that’s AI-complete. Having this kind of filter on your communications lets you monitor far more channels than you could without that help.

Companies like AideRSS are doing this already with their PostRank algorithm to pick out relevant articles from the noise of the blogosphere. You could even view Google’s PageRank as doing something similar for the wider web, and Xobni attempts to do the same thing for email. The key is that these are largely statistical methods that don’t need to understand the contents of a document at all.

Implicit Data

A lot of Vinge’s writing involves some kind of intelligence agency as a protagonist, and the use of advanced traffic analysis to reveal hidden information. Both traffic analysis and Google’s PageRank are part of a class of algorithms that use incidental, ‘implicit’ data to infer information that can’t be accessed directly.

Me.dium is trying to do something practical with this, building tools like search on top of massive amounts of data gathered on their user’s browsing habits, though you can argue that companies like Amazon have been doing it for years with their proprietary recommendation systems. The biggest barrier to wider adoption is how hard it is to access information across services or companies. To realize the promise of implicit data, we’ll need a lot more openness across the different silos that currently exist.