A few weeks ago I was lucky enough to have the chance to present at the Linley Processor Conference. I gave a talk on “What TinyML Needs from Hardware“, and afterwards one of the attendees emailed to ask where some of my numbers came from. In particular, he was intrigued by my note on slide 6 that “Expectations are for tens or hundreds of billions of devices over the next few years“.

I thought that was a great question, since those numbers definitely don’t come from any analyst reports, and they imply at least a doubling of the whole embedded system market from its current level of 40 billion devices a year. Clearly that statement deserves at least a few citations, and I’m an engineer so I try to avoid throwing around predictions without a bit of evidence behind them.

I don’t think I have any particular gift for prophecy, but I do believe I’m in a position that very few other people have, giving me a unique view into machine learning, product teams, and the embedded hardware industry. Since TensorFlow Lite Micro is involved in the integration process for many embedded ML products, we get to hear the requirements from all sides, and see the new capabilities that are emerging from research into production. This also means I get to hear a lot about the unmet needs of product teams. What I see is that there is a lot of latent demand for technology that I believe will become feasible over the next few years, and the scale of that demand is so large that it will lead to a massive increase in the number of embedded devices shipped.

I’m basically assuming that one or more of the killer applications for embedded ML become technically possible. For example, every consumer electronics company I’ve talked to would integrate a voice interface chip into almost everything they make if it was 50 cents and used almost no power (e.g. a coin battery for a year). There’s similar interest in sensor applications for logistics, agriculture, and health, given the assumption that we can scale down the cost and energy usage. A real success in any one of these markets adds tens of billions of devices. Of course, the technical assumptions behind this aren’t certain to be achieved in the time frame of the next few years, but that’s where I stick my neck out based on what I see happening in the research world.

From my perspective, I see models and software already available for things like on-device server-quality voice recognition already, such as Pixel’s system. Of course this example currently requires 80 MB of storage and a Cortex A CPU, but from what I see happening in the MCU and DSP world, the next generation of ML accelerators will provide the needed compute capability, and I’m confident some combination of shrinking the model sizes and increased storage capacity will enable an embedded solution. Then we just need to figure out how to bring the power and price down! It’s similar for other areas like agriculture and health, there are working ML models out there just looking for the right hardware to run on, and then they’ll be able to solve real, pressing problems in the world.

I may be an incorrigible optimist, and as you can see I don’t have any hard proof that we’ll get to hundreds of billions of devices over the next few years, but I hope you can at least understand the trends I’m extrapolating from now.