Edwin from Feedly kicked off a great discussion in the comments on Jeff's blog post about their suggestions for startup development. I actually really like their recipe, I'm a true believer in Customer Development and a stats fanatic, but I also know that metrics can be used for evil as well as good.

I'm worried that the important step is being glossed over; metrics are only as good as what you do with them. In Edwin's algorithm, the process and importance of measurement is gone into in detail, but the action you're supposed to take based on them is vague. What if "validate that the changes you are making are improving the metrics you are tracking" contradicts the later advice "you want … to try to polarize"? What if a change decreases average satisfaction but makes a minority ecstatically happy?

The default action would be to do more of things that raise your metrics, and less of those that lower them. In computer science we'd call this a greedy algorithm, and it may only lead to a 'local maximum' because it commits to choices too early. To picture this, imagine trying to get to the top of a mountain by only walking uphill. If the mountain were a perfect cone, you'd get to the top, but if it's more complex you may get stuck at the top of a much lower foothill.

For another analogy, imagine trying to walk to the north pole by always following a compass north. You'd hit a lot of obstacles in the way, and would probably have to take some excursions east, west or even south to get to your goal.

This isn't an abstract discussion for me; a lot of my previous work has been at large companies where innovation is stifled because metrics make it impossible to propose something that reduces any metric, even where there's massive customer benefit. For an extreme example of this, see Tom Evslin's experiences at AT&T:
http://blog.tomevslin.com/2005/02/att_lesson_from…

The
engineers were so fixated on the traditional "6 9's" of reliability as
a metric (meaning no more than one in a million calls could fail) that
they killed attempts to pioneer cell phones, internet access and VOIP.

I know from their great product that the Feedly team had to make hard choices to get where they are. What I really need are more war stories where founders had to take risks and do something that caused the metrics to drop in the short-term in the hope of long-term gain. Those are the decisions I find difficult!