All photos © Heather Champ, with permission

My biggest worry when I moved to San Francisco in December was that my dog Thor would find urban life tough, without the wide-open spaces we'd got used to in Colorado. I found an apartment with a wide sunny window-sill for him to lie on and next to the great off-leash Duboce dog park, but what I didn't realize until I started exploring was that there was another gem just half a mile away. The first signs I saw of Buena Vista Park were the trees at its peak looming over the neighborhood, their tops wrapped in fog. Following Duboce Avenue uphill to its end, and then meandering through 37 acres of woodland, I found myself 575 feet high, looking out over the city as the clouds cleared.

It's now become our morning walk, and we can make it to the peak and back in 45 minutes if I'm in a hurry. That's not often though, because catching up with the other regulars has become part of the pleasure. In the peculiar way of the dog-walking world, I often feel like I know the canines before I've properly met the owners, especially since I can't compete with Thor's natural charms. A great case in point is Bug and Chieka's owner, or as she's better known in the tech world, Heather Champ, the pioneering former Flickr community manager who furnished these photos.

Though it's not on the same scale as Golden Gate or Presidio, it's actually the oldest official park in the city, and is full of corners and history to explore. It was only when I was chatting to one of the gardeners that I realized the marble chunks lining some of the paths were actually fragments of gravestones from the cemetery that was razed by WPA workers in the 30's. At night its nooks can make it less welcoming, with hoboes camping out and casual hookups, but in the morning it's a slice of heaven.

If you live anywhere near the Lower Haight, Upper Castro or Noe Valley areas, you really should check out this urban garden. There's nothing like wandering through groves of Eucalyptus and Redwoods, watching the fog blow through the canopy, to refresh your soul after working through painfully obtuse YouTube comments (just to pick a random example, ahem). And if you see a cute Chihuahua mix with a snaggle-tooth, be sure to make a fuss of him.

I just discovered Maria Scileppi’s Living Brushstroke project that uses iPhone location tracking to create artistic views of people’s movements. The picture above is from the Chiditarod, “Probably the world’s largest pub crawl/food drive”. Almost everyone’s first reaction to seeing the trace of their movements retrieved from the iPhone logs is the same as ours, “Cool!”. If we can give users control over their own data (asking permission to record as Maria’s app does), there’s so many amazing projects like this we can build.

Anthem from Maria Scileppi on Vimeo.

Photo by Evil Science Chick

Some of you may have noticed a weekend hack I put together with Alasdair Allan for visualizing location data on your iPhone. Here's some random links related to the project:

Tell-All Telephone – View a German politician's life as a visualization, after he agreed to have detailed recording software added to his phone. [Update, I misread the story, and it was actually information gathered without his knowledge, but that he agreed to share afterwards to raise awareness. As commenter Seve says "The movement data of the german politician Matle Spitz were collected by law and not with his agreement. The movement and call data of everybody connected to a german mobil phone network were stored for 6 months. Finally the german supreme court stopped that law and the data were deleted. "

Geoloqi – A fascinating system that lets you volunteer to track your own movements, and share certain aspects of them with people you trust.

Location tracking on Android too – Detailed and fair technical analysis of the way that Android monitors your location, and how it differs from Apple's approach.

A Cryptographic Approach to Location Privacy – There are ways to get a lot of the benefits of location services without recording or revealing your position. Arvind's proposal shows how one application could work in a secure way.

22 Free Tools for Data Visualization and Analysis – The navigation is tough without a table of contents, but this is a guide overview of a lot of the tools you can use to turn your data into a visual story.

Photo by Wonderlane

I've just completed deploying a new version of the toolkit. It contains quite a few bug fixes and improvements, along with two new features:

UK Support 

You can now enter British postal addresses into the street2coordinates geocoder, and it will return the coordinates. With post codes included, it's normally accurate to within a couple of hundred feet. You can then run those positions through coordinates2politics to get the parliamentary constituency, county, council district and ward, NHS area and post code.

Time and date extraction

The new text2times method will scan through the text that you pass it, and pull out any strings that it can understand as times or dates. These include both a variety of formal date/time combinations like '10/28/01' as well as informal descriptions like 'next Friday'.

You can try it out by going to http://www.datasciencetoolkit.org/ , the command-line tools are at http://www.datasciencetoolkit.org/python_tools.zip , the new AMI is ami-f6e11d9f, and there's a new VM at http://static.openheatmap.com/dstk_v0.35.vmwarevm.tar.bz2

You should still be able to use your existing code with no changes, I've done my best to ensure everything's backwards-compatible, but let me know if anything breaks.

Photo by Nicolas Suzor

Newscoop – A content-management system designed by and for journalists. It’s been used in conjunction with Ushahidi to interesting effect.

Crisismappers extends UN capacity in Libya – As the power of crowd-mapping becomes more obvious, there will be pressure to use it in more ambiguous situations than natural disasters or the toppling of tyrants. Mapping military airbases in Libya gives a hint of what crowd-sourced warfare could look like.

StarCluster – A simple way to run and manage clusters of EC2 machines for scientific computing, complete with AMIs pre-loaded with useful software and sensible defaults. If this is your thing, you should check out Infochimp’s impressive ClusterChef too.

Fathom – A design firm with a portfolio of clear, crisp and beautiful infographics,

Trunkler – Link curation for your iPhone, built on top of the powerful Trunk.ly service. Shows why having a third-party API can pay off, even in the early days.

Photo by Joshua Mellin

A few times over the last few weeks I've been talking to friends about big tech company news, and one of the hardest struggles I have is to convince them that the latest Twitter or Google happening could just be a random cockup, rather than a sign of hidden plans by the company's management. I could well be wrong, but Google releasing a group-messaging app that doesn't run on Android reminds of the time I was involved in launching a GPU image processing API at Apple that competed against a similar interface created by another team at almost the same time. We mostly managed to keep that sort of thing from reaching the outside world, but every team and department in a large corporation is competing against all the other teams for a slice of the budget pie, and can have very different goals. Good management will keep the worst excesses in check, but any large organization is a massively distributed system where the communication overhead of keeping everyone totally in line would be crippling.

I was sorry to learn that large dinosaurs are no longer thought to have had an extra brain in their buttocks, as I'd memorably learned as a child from the Dinosaur Club. It's still a pretty good image for the decision-making apparatus in big companies though. Upper-management's most precious resource is time, and so attention has to be rationed. Especially at a company like Google that prizes experimentation, that means lower-level people can release projects into the wild that don't fit into the grander strategy. As another example from Apple, I know several teams that still hadn't ported their code over to the Cocoa framework even by the time I left in '08. Eating your own dog food is a fine goal, but if it comes down to that or shipping, sometimes pragmatism wins.

Think of my approach as the cock-up theory of technology news. The next time one of the big firms does something that makes no sense at all, consider taking it at face value. As General Nasser said in the 50's – "The genius of you Americans is that you never make clear-cut stupid moves, only complicated stupid moves which make the rest of us wonder at the possibility that we might be missing something".

Photo by Daniela Hartmann

I was talking to someone recently who at first I thought was an angel investor interested in my work, but it turned out he was keen on funding charitable projects. It shows I need to improve my messaging – while I am technically running a non-profit, that's not by choice. I do love supporting that world, but I'm always looking for revenue opportunities where I'm adding value for users who can afford it, even for my open-source projects.

It did get me thinking about some of my favorite non-profit teams though. They're all primarily focused on using data to do good, even if they're not purely non-profit in structure. Here's the people I'd trust to put my money to good use, if I was a social investor.

Ushahidi/SwiftRiver

With uptake worldwide, Ushahidi has been doing valuable work spreading information around disasters and political unrest, everywhere from Egypt to New Zealand and Japan. The SwiftRiver part is the brainchild of Jon Gosier, and while I'm a bit biased as I'm contributing some time and code, it's a really powerful use of technologies that are usually only focused on ad-serving. There's so many practical ways this kind of filtering and routing of information can help in an emergency.

Global Virus Forecasting Initiative

With feet on the ground in 23 countries, the GVFI has roots in the traditional non-profit world, but a clear understanding of how modern data processing can help its mission. Lucky Gunesekara, Lalith Polepeddi and the rest of their team are building a cutting-edge information infrastructure to make sense of the masses of information gathered by the teams, matching it against social media and traditional news organizations. The results are then used to track and predict human viral outbreaks and save lives. 

Media Cloud

Ethan Zuckerman is a legend in the non-profit world as a driving force behind projects like Global Voices and Geekcorps, but with Media Cloud he's applying his skills to data analysis on an ambitious scale. As the Arab revolts show, one of the biggest problems around the world is the way people are denied a voice. By tracking and quantifying media coverage in unprecedented detail, the project hopes to draw a picture of the problem, as a first step to offering solutions. It's an important project in a lot of ways – just ask yourself when you first heard about the Tunisian uprising? It took three weeks for coverage to enter most of the Western media (though France was a bit ahead of the curve, thanks to their colonial history). Those sort of gaps in our collective vision mean we're missing out on vital information.

Picture by Tiger Pixel

How US News abandoned print and learned to love its data – How a magazine started monetizing its rankings of colleges, cars, high schools and mutual funds as a side-business, and ended up closing the traditional publishing business to focus on it exclusively. People in the publishing business often think they’re selling books, magazines or shows, but those are just the delivery mechanisms for the advice and entertainment people actually crave.

eq.org.nz – The power of Ushahidi – How the local online community responded to the Christchurch earthquake. The details in this account are crucial, especially how important the ‘neutral ground’ aspect of the service was. With no corporate logos on the site, competitors felt it was safe to cooperate without worrying that it would backfire on them. That attitude may seem crazy in the face of a catastrophe, but efforts like this are way more effective when they don’t require people to behave like saints.

Article text extraction from HTML documents – In-depth bibliography on all the projects out that take a web page, and try to extract the important “body” text, without the ads, boilerplate or navigation links.

Big Data, Analytics and Storytellings – I spent an hour chatting with Lyle Wallis last week, he’s spent years fighting in the trenches applying data analysis to real-world problems. He had a lot of insights, but this post captures one of the most obvious but also most overlooked: People respond to stories. As engineering-types, it’s easy to miss out on the power of laying out your explanation as a narrative, but it’s a marvelous mental hack for connecting with your audience. See also Ira Glass.

Mobile internet usage in Japan – Comscore does a great job of teasing out the effects of the quake on cell data traffic. It actually surprised me how subtle the patterns of the disaster were in the data, even through something that apocalyptic.

Photo by Bill Bradford

Websockets Pacman – My friend Tyler Gillies created this basic but functional networked version of the classic game, where visitors to the site each take control over one of the ghosts. Here’s the playable demo. I love this because it shows how far our tools have come – just a few years ago it would have been a major engineering challenge, now it’s just a screenful of code.

The Anti-Predictor – Part of a fascinating interview with “Mathematical Sociologist” Duncan Watts, where he lays out the evidence against the existence of an elite of influencers, at least as most online marketers understand the term. “It does matter, on average, how many followers you have and how successful you’ve been in spreading your messages in the past, but it’s a lot more random than intuition suggests.”

The ethics of live mapping in repressive regimes and hostile environments – A detailed practical guide for online revolutionaries. The mundane detail of the precautions brings to life what enormous risks they’re running.

Componentization and Open Data – The data world is following the path that the coding world took a decade ago, as common foundational building-blocks start to become freely available. Certain parts, like address lookup, can be packaged as free and open components, which then allows engineers to work on the unsolved problems that really add value (and that you can charge money for).

FBI wants public help solving encrypted notes from murder mystery – Are you a cryptography geek? Donate some spare brain cycles to helping out the FBI.

A friend was asking me about a dog-walk I've been doing a lot on the weekends, heading up to the large antenna on top of Twin Peaks. Since it took me a little experimentation to figure out the best route, I've put together this little map and guide. It's a five mile round-trip, with some serious San Francisco hill climbing, and it's almost entirely along streets. It rewards you with some amazing views all the way though, especially from the summit. It usually takes me and Thor a little under two hours, so it's a great way to squeeze a good workout into a busy day.

I start off from my apartment at Church and Duboce, and walk a couple of blocks along Hermann, into the Duboce dog park. This is a fantastic diversion for both of us, there's almost always a good bunch of dogs and owners. After Thor's done his socializing, we then take Noe down to Market. Following Market for a couple of blocks takes you to 17th Street. This goes straight up, and keeps teasing you with false summits. Eventually, after a tough climb, you'll turn onto Clayton.

This turn is the only slightly tricky part of the route, since you take Clayton for a block, and then cross over to Twin Peaks Boulevard as it forks off. Going up past the small Tank Hill park, you'll then continue along Twin Peaks as it cuts left, away from Clarendon. This is the final stretch, and only the first section has any sidewalk. After that, you're walking on the shoulder, and while it's not too tough, be careful of the corners with poor visibility if you have a dog who thinks he should be able to walk down the center line. Some of the downhill bikes and cars pick up quite a lot of speed. There are social trails cutting a lot of the corners, so I'd definitely consider those.

Finally you'll make it up to the viewpoint. Especially after a rain, you get a magnificent vista, stretching out over downtown, across the Bay, and into Marin. Take it all in, and try not to be too disturbed by the posters asking for information on a recent homicide in the parking lot. I'm guessing it's quite a different world at night, but during the day it's full of visitors enjoying the sights.