A lot of people on this morning’s webcast asked if I had an Amazon EC2 image of pre-installed Caffe. I didn’t then, but I’ve just put one together! It’s available as ami-2faaa96a in the Northern California zone. There’s also a Vagrant VM at https://d2rlgkokhpr1uq.cloudfront.net/dl_webcast.box, and I’ve got full instructions for setting up your own machine on the Caffe wiki. I’m shaving yaks, so you don’t have to!
I’ve been having a lot of fun working with the O’Reilly team on ways to open up Deep Learning to a wider audience of developers. I’m working on a book tentatively titled “Deep Learning for Hackers” (since I’m a fan of Drew and John’s work), I’ve put up some introductory chapters as articles on the O’Reilly website, and tomorrow I’ll be doing an hour-long webcast walking through the basics of training and running deep networks with Caffe.
I hope you can join me, it’s a big help whenever I can collaborate with an audience to understand more about what developers need! If you do, I recommend downloading the Vagrant box ahead of time to make it easy to follow along. I look forward to seeing you there.
Databending using Audacity effects - What happens when you apply audio effects to images? Occasionally-wonderful glitch art! It always lifts my heart to see creative tinkering like this, especially when it’s well-documented.
Scan Processor Studies - On the topic of glitch art, here are some mind-bending analog video effects from 1972. They look amazingly fresh, and I discovered them while following the influences behind the phenomenal Kentucky Route Zero, which features a lot of tributes to the Bronze Age of computing.
The crisis of reproducibility in omen reading - In which I cheer as the incomparable Cosma Shalizi takes on fuzzy thinking, again.
What’s next for OpenStreetMap? - Has an essential overview of the Open Database License virality problem that’s stopped me from working with OSM data for geocoding. David Blackman from Foursquare gave a heart-breaking talk describing all the work he’s put into cleaning up and organizing OSM boundaries for his open-source TwoFishes project, only to find himself unable to use it in production, or even recommend that other users adopt it!
Where is my C++ replacement? - As a former graphics engine programmer, I recognize the requirements and constraints in this lament. I want to avoid writing any more C or C++, after being scared straight by all the goto-fail-like problems recently, and Go’s my personal leading contender, but there’s still no clear winner.
When I saw Cate Huston’s latest post, this part leapt out at me:
‘There’s often a general disdain for front-end work amongst “engineers”. In a cynical mood, I might say this is because they don’t have the patience to do it, so they denigrate it as unimportant.’
It’s something that’s been on my mind since I became responsible for managing younger engineers, and helping them think about their careers. It’s been depressing to see how they perceive frontend development as low status, as opposed to ‘real programming’ on crazy algorithms deep in the back end. In reality, becoming a good frontend developer is a vital step to becoming a senior engineer, even if you end up in a backend-focused role. Here’s why.
You learn to deal with real requirements
The number one reason I got when I dug into juniors’ resistance to frontend work was that the requirements were messy and constantly changing. All the heroic sagas of the programming world are about elegant systems built from a components with minimal individual requirements, like classic Unix tools. The reality is that any system that actually gets used by humans, even Unix, has its elegance corrupted to deal with our crazy and contradictory needs. The trick is to fight that losing battle as gracefully as you can. Frontend work is the boss level in coping with those pressures and will teach you how to engineer around them. Then, when you’re inevitably faced with similar problems in other areas, you’ll be able to handle them with ease.
You learn to work with people
In most other programming roles you get to sit behind a curtain like the Great and Powerful Wizard of Oz whilst supplicants come to you begging for help. They don’t understand what you’re doing back there, so they have to accept whatever you tell them about the constraints and results you can produce. Quite frankly it’s an open invitation to be a jerk, and a lot of engineers RSVP!
Frontend work is all about the visible results, and you’re accountable at a detailed level to a whole bunch of different people, from designers to marketing, even the business folks are going to be making requests and suggestions. You have nothing to hide behind, it’s hard to wiggle out of work by throwing up a smokescreen of jargon when it’s just changing the appearance or basic functionality of a page. You’re suddenly just another member of a team working on a problem, not a gatekeeper, and the power relationship is very different. This can be a nasty shock at first, but it’s good for the soul, and will give you vital skills that will stand you in good stead.
A lot of programmers who’ve only worked on backend problems find their careers limited because nobody wants to work with them. Sure, you’ll be well paid if you have technical skills that are valuable, but you’ll be treated like a troll that’s kept firmly under a bridge, for fear you’ll scare other employees. Being successful in frontend work means that you’ve learned to play well with others, to listen to them, and communicate your own needs effectively, which opens the door to a lot of interesting work you’d never get otherwise. As a bonus, you’re also going to become a better human being and have more fun!
You’ll be able to build a complete product
There are a lot of reasons why being full-stack is useful, but one of my favorites is that you can prototype a fully-working side-project on your own. Maybe that algorithm you’ve been working on really is groundbreaking, but unless you can build it into a demo that other people can easily see and understand, the odds are high it will just languish in obscurity. Being able to quickly pull together an app that doesn’t make the viewer’s eyes bleed is a superpower that will make everything else you do easier. Plus, it’s so satisfying to take an idea all the way from a notepad to a screen, all by yourself.
You’ll understand how to integrate with different systems
One of the classic illusions of engineers early in their career is that they’ll spend most of their time coding. In reality, writing new code is only a fraction of our job, most of the time will go into debugging, or getting different code libraries to work together. The frontend is the point at which you have to pull together all of the other modules that make up your application. That requires a wide range of skills, not the least of which is investigating problems and assigning blame! It’s the best bootcamp I can imagine in working with other people’s code, which is another superpower for any developer. Even if you only end up working as a solo developer on embedded systems, there’s always going to be an OS kernel and drivers you rely on.
Frontend is harder than backend
The Donald Knuth world of algorithms looks a lot like physics, or maths, and those are the fields most engineers think of as the hardest and hence the most prestigious. Just like we’ve discovered in AI though, the hard problems are easy, and the easy problems are hard. If you haven’t already, find a way to get some frontend experience, it will pay off handsomely. You’ll also have a lot more sympathy for all the folks on your team who are working on the user experience!
When I chat to people about deep learning, they often ask me what I think its future is. Is it a fad, or something more enduring? What new opportunities are going to appear? I don’t have a crystal ball, but I have now spent a lot of time implementing deep neural networks for vision, and I’m also old enough to have worked through a couple of technology cycles. I’m going to make some ‘sophisticated wild-ass guesses’ about where I think things will head.
Deep learning eats the world
I strongly believe that neural networks have finally grown up enough to fulfil their decades-old promise. All applications that process natural data (speech-recognition, natural language processing, computer vision) will rely on it. This is already happening on the research side, but it will take a while to percolate fully through to the commercial sector.
Training and running a model will require different tools
Right now experimenting with new network architectures and train models is done with the same tools we use to run the models to generate predictions. To me, trained neural networks look a lot like compiled programs in a very limited assembler language. They’re essentially just massive lists of weights with a description of the order to execute them in. I don’t see any reason why the tools we use to develop them, which we use to change, iterate, debug, and train networks, should be used to execute them in production, with its very different requirements around interoperability with existing systems and performance constraints. I also think we’ll end up with small numbers of research-oriented folks who develop models, and a wider group of developers who apply them with less understanding of what’s going on inside the black box.
Traditional approaches will fight back
Deep learning isn’t the end of computer science, it’s just the current top dog. Millions of man-years have gone into researching other approaches to computer vision for example, and my bet is that once researchers have absorbed some of the lessons behind deep learning’s success (eg use massive numbers of training images, letting the algorithm pick the features), we’ll see better versions of the old algorithms emerge. We might even see hybrids, for example I’m using an SVM as the final layer of my network to enable fast retraining on embedded devices.
There will be a gold-rush around production-ready tools
Deep learning eating the world means rapidly growing demand for new solutions, as it spreads from research into production. The tools will need to fit into legacy ecosystems, so things like integration with OpenCV and Hadoop will become very important. As they get used at large scale, the power and performance costs of running the networks will become a lot more important, as opposed to the raw speed of training that current researchers are focused on. Developers will want to be able to port their networks between frameworks, so they can use the one that has the right tradeoffs for their requirements, rather than being bound to whatever system they trained the model on as they are right now.
What does it all mean?
With all these assumptions, here’s where I think we’re headed. Researchers will focus on expanding and improving the current crop of training focused libraries and IDEs (Caffe, Theanos). Other developers will start producing solutions that can be used more widely. They’ll be able to compete on ease-of-use, performance (not just raw speed, but also power consumption and hardware costs), and which environments they run in (language integration, distributed systems support via Hadoop or Spark, embedded devices).
One of the ways to improve performance is with specialized hardware, but there are some serious obstacles to overcome first. One of them is that the algorithms themselves are in flux, I think there will be a lot of changes over the next few years, which makes solidifying them into chips hard. Another is that almost all the time in production systems is spent doing massive matrix multiplies, which existing GPUs happen to be great at parallelizing. Even SIMD instructions on ordinary CPUs are highly effective at giving good performance. If deep networks need to be run as part of larger systems, the latency involved in transferring between the CPU and specialized hardware will kill speed, just as it does with a lot of attempts to use GPUs in real-world programs. Finally, a lot of the interest seems to be around encoding the training process into a chip, but in my future, only a small part of the technology world trains new models, everyone else is just executing off-the-shelf networks. With that all said, I’m still fascinated by the idea of a new hardware approach to the problem. Since I see neural networks as programs, building chips to run them is very tempting. I’m just wary that it may be five or ten years before they make commercial sense, not two or three.
Anyway, I hope this gives you some idea of how things look from my vantage point. I’m excited to be involved with a technology that’s got so much room to grow, and I can’t wait to see where it goes from here!
Crossing the great UX/Agile divide – Methodologies are driven by human needs, and I know from personal experience that agile development takes power from designers and gives it to engineers. That doesn’t mean it’s wrong, but admitting there’s a power dynamic there makes it at least possible to talk about it candidly. “Although many software developers today enjoy the high salaries and excellent working conditions associated with white-collar work, it may not stay that way and UX design could be a contributing factor.“
Eigenmorality – A philosophical take on how PageRank-like algorithms could be used to tackle ethical dilemmas, featuring Eigenmoses and Eigenjesus.
The elephant was a Trojan horse – I almost always use Hadoop as a simple distributed job system, and rarely need MapReduce. I think this eulogy for the original approach captures a lot of why MapReduce was so important as an agent of change, even if it ended up not being used as much as you’d expect.
Neural networks, manifolds, and topology – There are a lot of important insights here. One is the Manifold Hypothesis, which essentially says there are simpler underlying structures buried beneath the noise and chaos of natural data. Without this, machine learning would be impossible, since you’d never be able to generalize from a set of examples to cope with novel inputs, there would be no pattern to find. Another is that visual representations of the problems we’re tackling can help make sense of what’s actually happening under the hood.
The polygon construction kit – Turns 3D models into instructions for building them in the real world. It’s early days still, but I want this!
I’m very proud and excited to be taking my oath of allegiance this morning, the final step to becoming a US citizen after thirteen years of calling this country my home. To mark the occasion, my girlfriend Joanne wanted to interview me to answer some pressing questions about exactly why I still can’t pronounce “Water” correctly!