I used to be an image processing engineer. I'd be handed a picture, and I'd have to do something useful with it. To do that I had to take a big mental leap. Instead of seeing it as an image I had to picture it in my mind as a grid of measurements.
At first this was intensely frustrating, because they were deeply crappy measurements. A million factors introduced noise or errors, everything from lenses to sensor noise to encoding software. Gradually I began to make progress, despite all these problems. Decades of engineers before me had figured out inelegant but effective methods of getting value from an unpromising soup of pixels, and I was able to learn from their approaches.
Interesting algorithms in image processing are almost comically domain specific. Thousands of man years of work have gone into detecting and correcting the distinctive reflections that occur when peoples' eyes are caught in a camera flash. Compressing photos effectively requires an exhaustive knowledge of the human perception system, and very clear ideas of the likely subject matter for photos. The process behind facial recognition is a like a game of Mouse Trap, with a whole series of steps that have been empirically proven to work, but which could never have predicted from any theory.
The computer science I was taught at college grew out of mathematics, and assumed that you have a minimal set of clean inputs. Provability and understandability were prized values, and so messy ad-hoc algorithms were seen as dead ends, even if they worked for the problem at hand. Image processing taught me to value them instead, as long as they could be proven to work across the kind of inputs I was likely to encounter in practice.
Once I'd learned that, the world began to look very different. My image processing training gave me the mental tools to tackle problems that other people shied away from. If I have a large enough set of data, I know how to search for the signal, even if the noise is deafening. I'm happy to rely on correlations that aren't guaranteed to hold for all time, as long as I can test it holds in the cases I care about now, and have instrumentation to spot if the prediction stops working. I know that getting 80% of the way there and having a human fill in the blanks is often good enough.
I wasn't the only one to discover how effective this mindset can be, and it has come to be known as Data Science. It's an approach to solving problems that's light on elegance and heavy on pragmatism. It doesn't care about proofs but relies on experiments. Entirely new things are possible once you have massive amounts of data, so even if you're a grizzled old engineer like me and instinctively shy away from trendy new labels, give Big Data a try. Amongst all the marketing hype, there's some powerful techniques for building algorithms that have no right to work, but do.
Pete Warden's blog 