I had a tough meeting with an advisor this week. I was proudly showing off how we've managed to triple the amount of time that first-time users spend on Jetpac, when he interrupted. He wanted to know why he should care? It forced me to quickly run him backwards through our decision-making process, looking at why we'd chosen that as one of the numbers we wanted to improve. We'd started there because we noticed that our most successful users, those who enjoy the app enough to keep coming back, tend to interact with the app a lot on their initial visit. Users who take more actions spend more time on the app, the correlation has always been strong in our case, so time was a good approximation of how much they were interacting. That had became the goal, and I had been so focused that it took me a moment to reconstruct how we'd got there.
The dangerous part was that there were lots of ways we could keep users on the app longer without improving the experience at all, or even making it worse! Luckily we have a lot of different methods of understanding how the experience is holding up, from surveys, crowd-sourced user tests, and contacts with power users, but it's still a risk.
When I was in college, a lecturer who was a grizzled engineering veteran warned us "You'll start off wanting to measure what you value, but you'll end up valuing what you can measure". You need to have fresh eyes looking at how you're evaluating your own progress, not only so you avoid the more obvious problem of vanity metrics, but also so you don't follow your numbers down a rabbit hole. Any measure is only a projected shadow of reality. When somebody asks "So what?", you always need to be able to point to something in the outside world that gets better when the metric does!
Pete Warden's blog 