Defrag’s first day

Ink

I’m just wrapping up the first day of Defrag, and it’s been exhilarating. It really does feel like several days at most conferences packed into one. You can see some great in-depth coverage of everything on AltSearchEngines, Charles was live-blogging in front of me in the morning.

First up was David Weinberger, billed as "Everything is miscellaneous", but actually renamed on the fly to "What’s unspoken between us". He’s posted his own notes from the night before, but what really struck me was how useful it can be to step back from the technological bubble I live in, and use the wider world to get a fresh perspective on what we’re doing. He drew on the poetry of Rilke and Rabbinic teachings to explain both how much meaning is implicit in our world, and the hope that computers will gain a soul, through their close association with us.

Clay Shirky couldn’t make it to give the next keynote, so instead there was a round-table discussion loosely based around "Social intelligence". Jerry Michalski moderated, with Joshua Schacter of de.licio.us fame, JB Holston from Newsgator and JP Rangaswami, the CIO of BT. It was unstructured, but full of interesting nuggets, such as Joshua never using RSS, or the idea of deliberately avoiding automatic spam blocking so that the community steps in. What I found most relevant to my work is the descriptions of the existing ‘attachment culture’ in most companies, where collaboration is done by emailing around Word, Excel and Powerpoint files. Mailing lists are a related ubiquitous big collaboration tool.

The general tone, being tech visionaries, was pretty derisive about these approaches, but I’m a contrarian on this one. I think if you can’t understand why people are resisting moving to the latest techniques, you’re probably overlooking some important advantages to the traditional tools. I developed some of these thoughts in a later open space discussion about user acceptance. Compared to a wiki, emails have a much simpler and more explicit security protocol. You make a decision about exactly who sees what you send out, with a clear chain of accountability if one of those recipients decides to make it more widely available. On a wiki, the visibility is determined by somebody else in an opaque way, and it’s a lot harder to understand who’s to blame if something does get wider exposure than it should.

Michael Barrett from PayPal then gave a mini-talk, "A message of warning". He’s worried that we’re merrily building tools with little thought to security, and that we’ll end up like telnet, impossible to use in any situation that requires secrecy. It’s hard to argue with that idea, but it’s also hard to see what the solution is. I’ve yet to see a startup wow its investors with a security demo, and there’s never a easy time to devote resources to securing your software. It took years of bad publicity before even MS moved away from fudging security in favor of user-facing features.

I’d never been involved with an UnConference-style open space, and it was hard to choose amongst all the topics. I picked the user acceptance theme, and to my surprise found myself the only one without a collar. It seems like the number one cause of failure of new tools is lack of user acceptance, so I was expecting a lot more techies and small companies. Instead it seemed dominated by people looking from the business side of large companies and trying to work out how to sell their employees on these ideas.

Andrew McAfee
of HBS was one of the leading voices in the discussion, and one of his most interesting ideas was that the security argument used to block use of wikis internally is hogwash. I’m no stranger to bogus security concerns being used to veto change, but as I said above, I think there’s at least a kernel of truth in this case. One of the unspoken ideas behind Defrag is that ‘information wants to be free‘, and a corporation will be a better and more productive place if only we can enable wider sharing.

This is probably true on the macro scale, but on the front lines there’s both winners and losers. Someone whose power is based around being the holder of arcane knowledge will fight tooth and nail against this. As a more benign example, a line-manager may not want other departments taking up his star engineer’s time to answer questions, if it’s to the detriment of the project he’s responsible for. Fundamentally, people’s performance is usually judged by the progress they make on their own tasks, not the overall benefit they provide to the company as a whole, since that’s a lot harder to measure. It would be nice to see this change, but until it does, there’s going to be resistance to collaboration, and the security argument is a useful tool in that resistance.

At lunch-time, I ended up sharing a table with Doc Searls, JP Rangaswami and Andrew, which was pretty heady company.

I had another tough choice to make after eating, since the conference split into two tracks. I wanted to see the panel with Adam Gross and John Crupi, especially after yesterday’s discussion with John about how to do something better with email, but Charles Armstrong from Trampoline Systems was speaking in the other room. They’re a really interesting company living in the same space I’m interested in; analyzing email to automatically figure out relationships within an organization, and then doing something useful with that. Dawn Foster and Aaron Fulkerson from MindTouch joined Charles on a panel themed "Social networking the enterprise".

Charles started with a description of his background as an ethnographer, and the inspiration for his work coming from his study of the communication techniques used by small communities. He’s using the insights he gained to write tools that analyze email and IM data, and use it to find experts within a company, or visualize the way people actually communicate within the organization. These are both areas that I’m really interested in, and it was great validation of the opportunities in this area to see how many customers Trampoline had picked up.

Alex Iskold from AdaptiveBlue/ReadWriteWeb then gave a talk on "A look at structured attention". He focused on the benefit to users of being able to control and share their own activity streams. As a practical example, if NetFlix could access your Amazon buying information, it could provide a lot better recommendations. He was proposing a model where there was some central, company-agnostic
data-store that all the services contributed to and pulled from. I’ve long been convinced that this would be a big leap forward, and allow startups without their own user-activity logs to do really interesting things, but I have a hard time understanding how to persuade Amazon to give up their competitive advantage. 

Alex asked whether the big players would open up, and in the discussion at the end, I pressed him about what his answer was, and what we should do if they do keep saying no. He seemed cautiously optimistic that it’s possible to produce some client-side approaches instead, which is something I’m betting on too.

Dick Hardt gave hands-down the most entertaining talk of the day, packing in 450 slides in 12 minutes, on the topic of "Defragging identity". I really need to try something similar for one of my corporate presentations, just to keep everyone awake! He’d obviously practiced like crazy to get the talk spot-on, it was a virtuoso performance. The content was good too, a primer on the history and evolution of trust, and how it was all based on past behavior predicting what someone would do in the future. He took us from the village where you knew everyone’s past first-hand, to cities where you had to trust strangers. After urbanization, people turned to third-party institutions to provide certificates indicating past behavior, for example a doctor’s qualifications. In the same way, online we want identities that we can attach tokens demonstrating past behavior to. These may not be a single, monolithic identity for everyone, we may use different identities in different situations, such as online games.

After this quick talk, Esther Dyson stepped up to the plate with "Discussing attention". It was really useful to hear her perspective on targeting advertising using individual consumer’s behavior, as someone who’d been involved on the marketing side. She had just returned from some FTC hearings on the same topic, and proposed a solution that had definite resonance with Alex’s ideas. The proposal was that users get access to the composite profile information that services generate from on the raw click-streams and buying habits, as hey can do with credit reports. This would allow consumers to escape from being labeled in incorrect or insulting ways, the "My Tivo thinks I’m gay" problem. Esther didn’t have a fully-formed proposal, but it was an interesting approach, and she was looking for feedback and improvements from the audience at Defrag. It raised some questions about who actually owns that data, the user or the company that captured it. With my client-based bias, I’m still pretty convinced that we’ll never persuade those firms to open up, and we’ll need to run on the user’s own machine to give them more control of that information.

In "Customer reach versus vendor grasp", Doc Searls was on very similar territory. He’s rebelling against the constant obsessive measuring and pigeon-holing  that’s behind personalized marketing. He asked "Who here wants to be better targeted?", and only one brave soul stepped up and said they did. Doc used a Walt Whitman poem to drive home the uniqueness and irreducibility of every human out there. That led to the idea that we should be able to control how companies see us, with him using the term "Vendor Relationship Management" to describe his approach, in opposition to the traditional customer relationship management that’s run by the vendors. He’s taken up the challenge of actually creating something like this with Project VRM, aimed at producing some practical software and standards to implement this vision. One of the compelling ideas he threw out is reversing the usual passive data model, where vendors pull information about user’s desires, and instead allow people to broadcast something they need, and see who can come up with a product that matches those requirements.

As somebody said in the discussion at the end, I’d love to lock Doc, Esther and Alex up in a room for a few hours, and see where their visions of the future match, and where they clash.

Ross Mayfield of SocialText gave the final talk of the day, and I was intrigued by the title, "Things to do in Denver when your corporation’s dead". Unfortunately he switched to "Made of people" instead, so I’ll never know how he would have lived up to the original heading. Ross’s talk covered a lot of ground, talking about the Radiohead album sale model, and how that approach could be used with other businesses, SocialText’s search for a new CEO and pulling in your customer’s expertise. The common thread with all of these is the active engagement of people all over the world in achieving your goal.

This just covers the formal talks, but of course some of the most interesting conversations happened in the hallway. I had a great chat with JC Herz about the work I’m doing on graph visualization, gave a demo to Robert Reich from me.dium, and received a demo of HiveLive from Greg Schneider.

I’m looking forward to another great day tomorrow, especially JC’s and Matthew Hurst’s talks on social visualization.

Funhouse Photo User Count: 2,099 total, 75 active.

Event Connector User Count: 105 total, 5 active.

CollectiveX and JackBe

Collectivexlogo
After dinner, I popped down to the hotel bar to peruse their selection of martinis. I met Clarence and Joe, the CEO and CEO CTO of CollectiveX.com. I hadn’t seen it before, but it’s an interesting service for anyone who’s collaborating. It provides a central location for group organization, you can email group members, have a shared calendar, host forums, all in one place. I chatted a bit about the ideas behind Event Connector, and how they tied in.
Jackbelogo

I also met John and Marilyn Jessica from JackBe. I was very interested to hear from John about his company’s experiences selling to large organizations. Marilyn Jessica also gave me a (probably well-deserved) ear-full about bringing a book along to read at the bar. I think the key adjective was ‘pretentious’, and my only defense was that I’m English, and I read the back of cereal boxes if I don’t have anything better to do. I also received some definitely well-deserved flak about the amount of rubbish I keep in my wallet, which I know Liz will second!

Updated with corrections to the unusual dual-CEO structure I implied, (maybe a RAID approach to corporate management?) and my deepest apologies to Jessica, who’s name I completely misremembered! All-in-all, an argument against posting after returning from cocktail hour. As penance, here’s the photo of my wallet that Jessica requested I add, to prove her point:

Wallet

Defrag has arrived!

Defragbanner

I flew into Denver this morning, and even though Defrag doesn’t officially start until tomorrow, I’ve already had a couple of early meet-ups with some of the local folks. It was fun seeing Rob and Josh from EventVue in the flesh for the first time, and hearing about all their hard work. They’ve been running at full steam since May I hope they get a chance for a break soon.

I also made an interesting discovery; Denver has two Hyatt hotels just a couple of blocks from each other, the Grand Hyatt and the Hyatt Regency. I only found this out after I’d dropped my car at the Grand’s valet parking and tried to check in! Luckily I was able to make it to the right one without further misadventure.

Funhouse Photo User Count: 2,097 total, 111 active. Ticking up gradually, with some good weekend active numbers.

Event Connector
User Count
: 109 total, 4 active.

Beautiful data

Datavisualization

With the mass of raw data I’m getting from a couple of years of my own email, I’m looking around for a good way to turn that into information. A simple ranking of my closest contacts is a good start, but I want to also see how much of the real-life groupings between others can be revealed. I’m working on a basic force-directed graph implementation, but that still leaves a lot of display choices.

VisualComplexity.com is one of my favorite places to find inspiration. They’ve done a great job collecting some of the most striking methods of presenting graph data visually. I also enjoy the Data Mining blog. Matthew’s a great resource and he’s good at reminding me to focus on getting something useful from my visualizations, not just pretty pictures. He’s headed to Defrag, so I hope I’ll get a chance to say hello.

Funhouse Photo User Count: 2,042 total, 52 active. Steady growth, but a low active count.

Event Connector User Count: 106 total, 13 active. A miniature growth spurt over the last day or two, with a comparatively large number of engaged users.

Beetlejuice, Beetlejuice, Beetlejuice

Beetlejuice

When I was 16, I’d sneak into a University of Cambridge computer lab, with the desire to download Amiga mods and demos from ftp sites. This was pre-web, but I owned The New Hacker’s Dictionary (aka the Jargon File), and had been very excited by the concepts of email and Usenet. What blew me away when I actually tried them was that the person who wrote my assembler reference was on the ARM newsgroup and answering questions from mere mortals like me!

I was reminded of that yesterday when Matt Brezina from Xobni left a comment here after I reviewed their product. I’m old enough to remember a time before the internet, when people just couldn’t connect like that. Now there’s services that let you know when someone talks about something you’re tracking, and start a conversation. That’s what I find really interesting about working with a social graph, building tools that help people build relationships.

That computer lab also led to me living in an Alaskan treehouse for three months, but that’s a story for another post!

Funhouse Photo User Count: 2,003 total, 73 active. Broke 2000!

Event Connector User Count: 100 total, 5 active. Broke 100, which isn’t quite as exciting.

Competing mail graph services

Wrestler
My idea of deriving a social graph from your email messages isn’t new. In a simple form, sites like Facebook already pull your contacts from webmail sites to build a first draft of your friends list. Unfortunately, your contacts list is a poor map of your actual relationships.

SNARF is a Microsoft project from 2005 which analyzes your email in a much more sophisticated way, and uses this information to help you triage your messages. It has a flexible system of metrics, for example the number of emails you’d sent someone, to calculate an importance score for each contact. Emails from important people are displayed in a top priority area of the UI, away from the less important bacn.

Xobni is a more recent Outlook add-in that analyzes your messages. The slick UI gives you a lot of interesting ways to drill down into the relationships it finds. It gives you fast searching, raw stats on each contact’s mailing patterns, and lots more. Its automatic phone number extraction from messages and the display of attachments by sender look particularly useful.

Looking further afield, there’s some overlap with spam protection services. Spam guards attempt to exclude a particular class of email by analyzing messages. I don’t know of any that go beyond checking if you’ve sent someone an email before in their social analysis, but it seems like a natural direction for some of those companies to head in.

Yahoo and Google both have access to the raw information from millions of mail users, so if they see an advantage in this sort of mail-based social graph, they could create something really compelling. One interesting area is the APIs they offer, which might be enough for a third-party developer to at least show a proof-of-concept demo, though it would no doubt be against the ToS.

What I’ve yet to see if a really painful problem that is solved by any of these services. Xobni is the closest, but it still feels like a great set of additional features for Outlook, not a make-the-pain-stop solution that people will pay for.

I’m certain there are some great ways to solve problems using a mail-based social graph, I just need to find them!

Funhouse Photo User Count: 1,998 total, 73 active. Almost up to 2000 total,  with no work on it for the last month.

Event Connector User Count: 91 total, 5 active. A few new users still coming in through the directory, but no conference sign-ups.

Take a walk on the client-side

Desktop

I’ve been following Mozilla’s Prism launch with a lot of interest. One comment that caught my eye was from  Ryan Stewart, who believes "that the desktop isn’t dead, and that a hybrid approach is a successful way to go". There’s a lot of opportunities for really useful services that need a client-side component, and that’s key to my strategy. It’s the reason I’ve worked on Java Applets, Firefox and Internet Explorer extensions and now an Outlook plugin.

Web-based apps have a lot of advantages over traditional desktop apps:
No install needed! It’s an incredible advantage to be able to go to a URL and instantly start using the service. The installation process is a big barrier to people even trying your app, since they’ve been trained to know it takes minutes, involves answering questions they may not know the answer to, and worrying about their configuration.
Complete safety. There’s no worries about viruses or spyware, the only information it has access to is what you type in.
Use anywhere. You can use the app from any machine in the world, regardless of OS or configuration.
Up-to-date and comprehensive information. Since the information you’re accessing is pulled live from a server, it can be updated instantly, along with the application itself.
Easy to develop. You have a lot fewer variables to worry about with a server-based app. You know exactly what hardware and software the main code is running on. Keeping the output HTML cross-platform is a couple of orders-of-magnitude easier than doing the same for executable code.

There are some obstacles that I think will prevent pure web-based services from taking over the app world:
A limiting sandbox. To achieve security, web-pages can only pull data from their own servers, and can’t use any of the information already on the user’s machine, or any other web-services the user is signed onto, without an explicit agreement with the provider. This effectively stovepipes all information, and is the reason I’ve entered my list of friends on at least a dozen different services. I don’t see this changing, because it would require a majority of the browser vendors to implement a more subtle security policy than the current blanket same-domain-policy. Loosening security like that doesn’t seem likely.
Poor UI. Google has done an astonishing job with their mail interface, but for power users the experience still doesn’t match Outlook or Apple Mail. Running within a web-browser makes it tough to offer a rich interface, and you can’t use the standard OS metaphors. This is where Prism is interesting, because XUL actually offers some of the UI that HTML is missing, like sliders.

What I think is really interesting is the idea of combining the strengths of the two approaches. To start off, there needs to be a pure web interface to build up enough interest and trust for people to download a client extension that offers more features. That extension can then offer some tools to the app, like cross-domain access and native UI, but still keeping the bulk of the application running on the server. That keeps the big advantages of SAAS, such as ease of development, and online access to information, but allows a whole range of applications that aren’t possible now.

Funhouse Photo User Count
: 1,975 total, 77 active.

Event Connector User Count: 89 total, 1 active.

La Jolla Valley hike-in campground

Lizwithscrabble

Liz and I just got back from an overnight backpacking trip in the local mountains. We had a new tent and other gear we wanted to test-drive, and didn’t want to be too stuck if some of it didn’t work. With this in mind, we chose the La Jolla Valley campground, since the trailhead’s less than an hour from our house, and it’s only a 2.5 mile hike in.

As I mentioned in a previous post, this campground is the only one I know of in the Santa Monicas that doesn’t require reservations, and it’s rare to see anybody using it, so we felt pretty confident we would get a spot. The recent wildfires to the east probably kept people away too. We’d checked the weather and the red-flag status before-hand, along with asking a friendly wildland firefighter friend, and it looked like there was no danger.

We stopped by the booth at Thornhill Broome Beach when we arrived, paid $3 per-person to camp and picked up a parking pass. We then drove across the street to the day-use parking lot, where the trail starts. There’s two trails leading off the lot, the Ray Miller section of the Backbone, and La Jolla Canyon trail. To get to the campground, we had to hike 2.5 miles up the La Jolla trail, which is the one behind the yellow gate, that starts off like a fire-road.

You start off with a moderate uphill grade going along the canyon floor. After about half a mile you cross over the wash, and start heading towards a waterfall. Sadly, with the drought, there was no water left when we got there, but it’s still a nice spot to catch your breath. The trail heads up the side of the waterfall, and then starts switchbacking up the canyon wall. The next half-mile is probably the toughest part of the hike, with poor footing and some steep grades.

At around 1.2 miles, you’ll reach a fork in the trail. Keep to the right/eastern side at this fork to get to the campground. The left/western trail heads towards Mugu peak, and connects with some of the other trails criss-crossing the valley to form some fun loops.

The trail is still uphill, but with a decent grade, and goes through some nice shady brush archways. 1.8 miles in, you get to another fork, this one with a new sign:
Lajollasign

It’s hard to tell from the sign, but you actually want to head left/west for the quickest route to the campground. It’s a short connector that skirts a pond, and the first camping spots are just off to the west of it. We didn’t investigate these, we knew there were some more further on, and the way to them looked a bit rough. The trail reaches a fire-road, and there’s camp spots just off this road, a short distance west. There’s no map of the camping spots, so you’ll just need to look for the numbered signs and explore.

We went to site 5, which was secluded, in a small oak grove. There’s no camp-fires or water, but they do have a pit-toilet nearby, and gas-stoves are allowed. Slightly further west along the fire-road is a group camp site, that looks like it could handle up to ten tents comfortably. We did notice the restroom had been knocked off its foundation by a fallen tree, so you’ll need to use the toilet that’s near the individual sites instead.

It was a pretty gruelling hike, we’d brought too much water, and there’s about 600 feet of elevation gain, so we were very happy to sit down at the picnic table and rest our feet. It was actually raining for once, so we got our new tent up as quickly as we could, to keep our stuff dry. I’m very happy with the REI Half Dome tent we got to replace our heavier car-camping one. It was only $99 on sale, but it weighs less than 5 pounds, was easy to put up and kept us dry, which is pretty much all I could ask for.

I saved money on the tent, but more than made up for it with the stove. I fell for the Jetboil a few weeks ago when a friend showed me his on Santa Cruz, and it was so worth it’s $130 price tag. Very light and compact, it heats water up incredibly fast and efficiently, has electric ignition, and is almost immune to wind.

With everything set up , we sat down to a meal and a serious game of Scrabble, as you can see in the lead photo. After some nice instant miso soup and an entree from Mountain House, we closed off the night with some martinis, served in genuine camping martini glasses:
Campingmartini

During the night we were serenaded by coyotes, some very close, and there were smaller varmints snuffling in the nearby bushes. There was nobody else camping in La Jolla, so it was a bit spooky to be the only people for miles, especially with a full-moon a few nights before Halloween, but we both slept well.

It was a wonderful trip, topped off with a 5 mile hike to Mugu Peak the next morning. If you’re looking for a beautiful camping spot only 45 minutes from LA, I highly recommend La Jolla Valley. Call Point Mugu State Park on
805-488-5223 for more information, and for other Santa Monica Mountains campgrounds, check out these posts:
La Jolla Valley
Sycamore Canyon
Circle X
Santa Cruz Island

Funhouse Photo User Count: 1.954 total, 77 active. Still slowly but steadily growing.

Event Connector User Count: 88 total, 1 active. I haven’t been focusing on selling this idea to any organizers for the last week, and it needs some time and effort to persuade promoters to consider it.

PNG loading made simple

Emptyframe
I need to load some images from disk, and use them as OpenGL textures. This is something I’ve reimplemented many times on different systems, and it’s always seemed more complicated than it should be.

The first choice is whether to use the native OS’s image loading libraries, or a cross-platform framework. Native libraries are a lot easier to start using, but make the code harder to port to other systems. Using an open-source framework makes your code much more portable, but it’s often a pain to find or build against the right version of the code, and you also have to deploy it to the end-users’ systems. I lost several hours wrestling with libpng a few weeks ago, since it refused to build on OS X, and I ended up yak-shaving my way down a dependency chain of other libraries it needed.

This seems overkill for most of the tasks I’m trying to accomplish, since I’m usually just trying to load a small, fixed set of images, which I’ve created and so can control their properties. This only uses a tiny part of most image loading libraries, since they’re designed to cope with reading and writing dozens of different formats, and hundreds of combinations of bit-depth, compression and other flags within each of those, along with lots of platform-specific optimizations to improve performance.

Happily, I recently came across the LodePNG project, which offers a simple, dependency-free way of loading PNGs. The PicoPNG module is a standalone C++ file that will load 8 bit, 4 channel PNGs, and is only 500 lines of code. This is perfect for my needs, there’s no extra installation hassles or dependencies, and it’s cross-platform. I also appreciate the elegant way that Lode Vandevenne packed his solution into such a small amount of code. It’s compact, but not obsfucated, very classy work.

Funhouse Photo User Count: 1,933 total, 78 active. Still getting quite a few users from the profile box add link, according to my statistics.

Event Connector User Count: 87 total, 4 active. Very quiet, I haven’t been actively chasing up any leads for a few days.

Jerry Mitcham, trailwork hero

Jerry

Last night was the annual Santa Monica Mountains Trails Council board meeting and dinner. Jerry received the Hank Grateful award for the most time worked this year, with an astonishing 366 hours spent fixing up the trails! For just one project, he hiked miles out into the backcountry with a 30 pound post hammer to install over 70 signs. He’s not only a hard worker, he’s also a leader, both day-to-day as a crew boss, and working with the park agencies to make work trips to Santa Cruz Island happen. A good friend and lovely guy, I always look forward to working with him out in the mountains.

He’s not alone out there, many of the regulars clocked up over 200 hours each. Looking around the room that night, I was thinking about the difference all these volunteers make, even though most trail users will never know about them. They’re not doing it for recognition, money or out of obligation, they just take joy in getting things done, in improving the mountains they love.

The awards had the motto "The reward for work well done is the opportunity to do more". The happiest people I know are those who love what they do. I love being in a team full of them, whether it’s in the mountains or a conference room.

Jerryandal