9 ms·

Why TinyML is still so hard to get excited about

123 points by mariuz 4y ago

reisender 4y ago
It seems like many of the TinyML use cases are hidden or somewhat dull, making it difficult to get people excited about it. Do you think there are ways to make TinyML more exciting and appealing to a wider audience? How can we better showcase the potential of TinyML and its ability to bring more privacy to IoT and give everyday products superpowers?
- hosh 4y ago
  How about just being able to run a home assistant with AI support to reprogram the home assistant, without having to spend a lot more electricity? They don’t have to have superpowers to be useful.
  - marcosdumay 4y ago
    I expect tinyML to be way too tiny for that. The clear use-case for it is mechanical control and feedback. But it seems that every robot has enough tiny problems that you can justify a larger CPU anyway. I too am having a hard time being excited about it.
  - STM32F030R8 4y ago
    My home assistant runs on 5-10W power. Nothing it does requires a lot more electricity than a light bulb.
    - hosh 4y ago
      Does that home assistant have a self-hosted LLM interface that helps you with configuration and setup?
vlovich123 4y ago
> So instead, Warden’s company is releasing a new sensor that can scan a QR code. The idea behind this $6 sensor is that appliance makers can put it inside their products as a method of getting devices onto Wi-Fi easier. A user could simply show their Wi-Fi QR code (I find mine in my router app) to the sensor and get their, say, fridge or washer online. I think it could be neat as a way to transfer a recipe to an oven, or specific washing instructions to a washing machine for particular items of clothing. Unfortunately, unlike scanning a new shirt and getting the machine to change its parameters to provide the best wash, many of the use cases for TinyML are going to be kind of boring. Sorry. What’s the ML piece here? QR codes are from the 90s and don’t use any ML I’m aware of… > However, at the conference Warden told me that, while he’d quickly discovered that the model worked, educating people about new gestures was tough. “No one knows that these gestures are available,” he said. This makes sense. If you remember back to the launch of the first iPhone and its touchscreen, the first ads and demonstrations focused on things like taps and pinch-to-zoom. Those weren’t intuitive; they were taught. And if that were true, that’s solvable just like with the iPhone by having tutorials when you boot your TV. I think what’s actually the case that the CEO doesn’t want to admit is that he’s having trouble convincing TV makers this is a useful model when they’re all going into voice-operated UIs. That and the BOM cost makes it unappealing.
- vlovich123 4y ago
  > Elsewhere at the event, HP showed off two TinyML implementations with ST Micro that are embedded in new laptops. The first TinyML model uses a gyroscope to detect if a laptop has been placed in a bag or taken out of a bag. The idea behind the implementation is that the laptop will start booting up when it’s taken out of a bag in preparation for its owner to use it. If the model detects the laptop has been placed in a bag, it will change heating and cooling parameters to make sure the laptop doesn’t overheat. And when the model gets it wrong (which it invariably does) you’ve got a laptop that failed to be in sleep in a bag (maybe you fallback to more primitive models that are foolproof like increased temp + lid closed = in bag). But seriously. If you go to sleep on lid close, putting it in a bag doesn’t really change your thermal envelope (should have happened on lid closed). And booting before your lid opens seems silly when Apple shows that it can be done near instantaneously. In other words, this seems like a PM developed feature for promo instead of good engineering being done. > The second use case also helps with thermal management. In that use case, the laptop detects when it is on a hard or soft surface. If it’s on a soft surface, like a bed or a person’s lap, it will try to run cooler so as to avoid overheating. Ok. Maybe this is interesting. But do you actually need ML or is it enough to define a thermal budget and recognize a solid surface can probably dissipate heat more quickly and anything beyond that doesn’t buy you all that much.
justaregulardev 4y ago
The ability to have models that can run on resource-constrained devices does feel like a strong direction for ML to go in and could lead to greater user privacy. However, I’m unconvinced by the IoT-aspect of this tech. In many ways, it feels like IoT has “failed” to be as popular with consumers as expected and feels overhyped. Will adding ML to IoT devices really make a difference?
- lamuswawir 4y ago
  IoT sensors powered by ML may not provide good use cases for consumers, mostly because all things they can do can be done by a large model in the cloud, plus a phone. It will get interesting when we ask what use cases can't be solved by phone+cloud combo. Such things as air quality management are good use cases. You can't use your phone to do that.
  - digging 4y ago
    > Such things as air quality management are good use cases. You can't use your phone to do that. Why not? I am strictly against IoT in my household so I may be way off base, but why can't your phone control your air purifier?
- mirker 4y ago
  Privacy doesn’t matter unless the IoT devices are secure. Often times, they’re not.
- hosh 4y ago
  Adding an embedded LLM as a human interface for every appliance is a huge win— for consumers at least. For example, I have a dishwasher with a bunch of settings, can sense load, etc. It’s got a touch interface that works with wet hands. Or I can tell it to start with the usual settings, or that a particular load is a bit different. Same with the laundry, the pressure cooker. It is less mind bandwidth when you got kids. What I don’t want, is for my appliances to do is to phone home to the makers. LLMs (if you don’t somehow trigger its insanity) can be far more capable than Siri. How do you get that into something more energy efficient than a high end gaming rig? Something more hidden is using LLMs to reprogram machine-to-machine protocols. That might extend the lifetime of machines that have to talk with other machines, but it breaks planned obsolescence. There are plenty of exciting product ideas. Whether they are exciting revenue generators are another thing entirely.
brrrrrm 4y ago
ML is still in a compute hungry growth phase. Every exciting new achievement in accuracy requires more compute than before. It makes sense that tinyML, which has less compute, would always have lagging modeling support and be a bit less interesting. TinyML does result in much cheaper products, though.
cxie 4y ago
TinyML would depend on the breakthrough in Edge/embed AI chips, which doesn't seem to come in the near future...
- jononor 4y ago
  Why do you say that? There are microcontrollers announced with neural network accelerators integrated, and low power FPGAs with accelerators on the market, along with many chip startups aiming to provide milliwatt scale dedicated neural chips.
hosh 4y ago
That’s weird. Being able to use AI without being beholden to Big Tech seems like a huge win to me. Especially when I want to ensure that the AI is making best effort for my local community’s best interest, and not the best interest of a large corp.
- mirker 4y ago
  Even so, you can run models on a private cloud. No need to do anything but sense on an embedded device.
  - hosh 4y ago
    That’s a huge burden on the natural resources (electricity). Beyond local control, I gave other examples of embedded AI in other comments — or more specifically, what you can do with an embedded LLM. Namely, being able to have a better human interface for complex settings, and being able to reprogram protocols (or anything that is “software-defined”) for future-proofing.
    - mirker 4y ago
      Integrating an accelerator into a dishwasher is also expensive. How many inferences are you going to get out of it before the hardware is outdated? Once a day for 5 years may not be enough to justify the cost. Compare that to having a single GPU in a server in your house that services all your IoT devices, which can be readily upgraded and requires no special software. The is no benefit to doing inference on device unless you are in the business of selling said devices.
      - hosh 4y ago
        A big part of TinyML is finding alternate paths for implementing AI in resource-constrained space. We already have SoC that is functionally not so much different from modern microcontrollers. Depending on economics of scale, it isn’t that big of a leap of imagination to see a microcontroller which includes 4-bit vector ops like a GPU.
    - 4y ago
RugnirViking 4y ago
From my work in AI & robotics, its generally finnicky and a bit crap even when a ton of work goes into it with a lot more compute (jetson nano or higher). I feel like trying to optimise those kinds of models is a bit niche/premature
- steve_adams_86 4y ago
  As a hobbyist that has been my perception of the situation as well, though I wasn’t sure if I was just missing something. The potential is immense and exciting, but try as I might, I can’t seem to make even fairly simple things work reliably as I want them to. One thing I love about embedded projects is that you can achieve pretty incredible reliability because everything can be so dialed in and isolated from points of failure in, say, an operating system. Trying to use ML for simple tasks felt like it eliminated that and introduced seemingly arbitrary failure into projects I really needed to work perfectly. Again, just a hobbyist, so I can’t make any broad statements. It seems to align with what you’re saying though. Once we get past this hump and have more powerful/effective models at a reasonable price point, I feel like it could be transformative. At the moment it’s still extremely interesting and fun to experiment with.
bambax 4y ago
I'm sure there are lots of fun things to do with an Arduino that could recognize gestures; I'm very surprised nobody's "excited" by this? For example, I made a photobooth simple device based on a Pi and a regular DSLR; the person takes a photo by pressing a button, then the Pi checks the camera and sends the latest image in a web gallery somewhere. The problem is the button: it needs a remote. If wired, it risks destroying the whole apparatus if someone pulls on the cord; if wireless, it risks being lost. A gesture-based trigger would be super cool; but having it run on the Pi through the DSLR risks damaging the camera, so it should run on a different device, cheap and not too power hungry, such as an Arduino. What about some kind of MIDI-controller based on an Arduino that could recognize gestures? I also made a webapp to learn sight-reading (babeloop.com) but it requires users clicking or tapping on the screen, which is not natural. It would be cool to be able to listen to users reading notes out loud and detect if they're right or wrong, on the fly, locally in a browser or on a phone. A light general speech recognition model such as VOSK is 40Mb and is able to recognize most phonemes; but for music sight reading, there are only 7 syllabes, so one should be able to make a much smaller model? I don't know how hard it would be though to train my own model...?
- JohnFen 4y ago
  > I'm sure there are lots of fun things to do with an Arduino that could recognize gestures Arduinos can recognize gestures, faces, etc. right now with well-established tech, no servers needed. I've had a couple of robots running around my place that do this sort of thing for a number of years now.
- tiedieconderoga 4y ago
  Embedded platforms generally have very little memory and very few hardware threads. Tiny SoCs will likely need accelerators to run more interesting models. For example, ARM is working on a ML coprocessor to pair with their Cortex-M chips. https://www.arm.com/products/silicon-ip-cpu/ethos/ethos-u55 https://www.arm.com/products/silicon-ip-cpu/ethos/ethos-u55
svantana 4y ago
I'm very into TinyML - not because of IoT but because small and fast models work well in (clientside) webpages, in realtime systems and pretty much everywhere. All while consuming less resources, which should be celebrated. Also, it's more democratic - anyone can compete in this field, supercomputer not needed. IMO, what is needed is a larger focus on resource consumption in competitions, benchmarks and rankings. For example, I like that you can rank some lists on paperswithcode by number of parameters [1]. [1] https://paperswithcode.com/sota/image-classification-on-imagenet?metric=Number%20of%20params&dimension=Top%201%20Accuracy https://paperswithcode.com/sota/image-classification-on-imag...
m3kw9 4y ago
Prob now you got GPT and there is a new baseline they have to live up to
Buttons840 4y ago
I've been learning how transformers work, and also reading about RNNs. It seems that transformers have succeeded because they can be parallelized, and solve many of the problems that RNNs have. This has lead to the advancements we've seen recently with GTP. Big companies are able to throw their data centers at the problem and train impressive and massive models. Part of me hopes that a non-parallelizable AI architecture might be discovered which performs even better. Perhaps the problems with RNNs could be solved in a way that doesn't parallelize? We would be so fortunate if a desktop computer could run an AI that's half as good as Microsoft's or Amazon's best AI. I would love to see the advantage of the data center removed. Philosophically, this does make some sense. The wisdom behind sayings such as "adding more people makes the project even later" exist because the greatest intellects we're aware of (ourselves) do not parallelize well.
bob1029 4y ago
After the initial obsession with Jupiter-scale models, we quickly discovered that very small models can perform even better if you reorganize your problem. Last week, we were trying to fine tune a 175b parameter model to take a natural language prompt with hopes of directly-outputting correct, domain-specific SQL. Now that reality has passed, we are looking at different paths. As of this week, we are trying to hit everything with the binary classification hammer. Turns out you can train a model to output 1 of 2 possible tokens with exponentially fewer parameters, training items, machine hours, etc. The statistics available in binary classification are also incredibly powerful and the results are trivial to reason with. Even if you need thousands of binary classifiers, their scale and granularity makes this a non-event or potentially an advantage. The real integration magic with AI/ML is starting to look like a weird form of set theory. At this level of complexity, detecting (and potentially confirming) the user's intention is way more important than trying to draw a direct map from input to destination.
tmtvl 4y ago
For anyone interested, it's not Meta Language ML (as in Standard ML), it's Machine Learning ML.

123 points by mariuz 4y ago

It seems like many of the TinyML use cases are hidden or somewhat dull, making it difficult to get people excited about it. Do you think there are ways to make TinyML more exciting and appealing to a wider audience? How can we better showcase the potential of TinyML and its ability to bring more privacy to IoT and give everyday products superpowers?

hosh 4y ago
How about just being able to run a home assistant with AI support to reprogram the home assistant, without having to spend a lot more electricity? They don’t have to have superpowers to be useful.
- marcosdumay 4y ago
  I expect tinyML to be way too tiny for that. The clear use-case for it is mechanical control and feedback. But it seems that every robot has enough tiny problems that you can justify a larger CPU anyway. I too am having a hard time being excited about it.
- STM32F030R8 4y ago
  My home assistant runs on 5-10W power. Nothing it does requires a lot more electricity than a light bulb.
  - hosh 4y ago
    Does that home assistant have a self-hosted LLM interface that helps you with configuration and setup?

vlovich123 4y ago

> So instead, Warden’s company is releasing a new sensor that can scan a QR code. The idea behind this $6 sensor is that appliance makers can put it inside their products as a method of getting devices onto Wi-Fi easier. A user could simply show their Wi-Fi QR code (I find mine in my router app) to the sensor and get their, say, fridge or washer online. I think it could be neat as a way to transfer a recipe to an oven, or specific washing instructions to a washing machine for particular items of clothing. Unfortunately, unlike scanning a new shirt and getting the machine to change its parameters to provide the best wash, many of the use cases for TinyML are going to be kind of boring. Sorry. What’s the ML piece here? QR codes are from the 90s and don’t use any ML I’m aware of… > However, at the conference Warden told me that, while he’d quickly discovered that the model worked, educating people about new gestures was tough. “No one knows that these gestures are available,” he said. This makes sense. If you remember back to the launch of the first iPhone and its touchscreen, the first ads and demonstrations focused on things like taps and pinch-to-zoom. Those weren’t intuitive; they were taught. And if that were true, that’s solvable just like with the iPhone by having tutorials when you boot your TV. I think what’s actually the case that the CEO doesn’t want to admit is that he’s having trouble convincing TV makers this is a useful model when they’re all going into voice-operated UIs. That and the BOM cost makes it unappealing.

vlovich123 4y ago
> Elsewhere at the event, HP showed off two TinyML implementations with ST Micro that are embedded in new laptops. The first TinyML model uses a gyroscope to detect if a laptop has been placed in a bag or taken out of a bag. The idea behind the implementation is that the laptop will start booting up when it’s taken out of a bag in preparation for its owner to use it. If the model detects the laptop has been placed in a bag, it will change heating and cooling parameters to make sure the laptop doesn’t overheat. And when the model gets it wrong (which it invariably does) you’ve got a laptop that failed to be in sleep in a bag (maybe you fallback to more primitive models that are foolproof like increased temp + lid closed = in bag). But seriously. If you go to sleep on lid close, putting it in a bag doesn’t really change your thermal envelope (should have happened on lid closed). And booting before your lid opens seems silly when Apple shows that it can be done near instantaneously. In other words, this seems like a PM developed feature for promo instead of good engineering being done. > The second use case also helps with thermal management. In that use case, the laptop detects when it is on a hard or soft surface. If it’s on a soft surface, like a bed or a person’s lap, it will try to run cooler so as to avoid overheating. Ok. Maybe this is interesting. But do you actually need ML or is it enough to define a thermal budget and recognize a solid surface can probably dissipate heat more quickly and anything beyond that doesn’t buy you all that much.

justaregulardev 4y ago

The ability to have models that can run on resource-constrained devices does feel like a strong direction for ML to go in and could lead to greater user privacy. However, I’m unconvinced by the IoT-aspect of this tech. In many ways, it feels like IoT has “failed” to be as popular with consumers as expected and feels overhyped. Will adding ML to IoT devices really make a difference?

lamuswawir 4y ago
IoT sensors powered by ML may not provide good use cases for consumers, mostly because all things they can do can be done by a large model in the cloud, plus a phone. It will get interesting when we ask what use cases can't be solved by phone+cloud combo. Such things as air quality management are good use cases. You can't use your phone to do that.
- digging 4y ago
  > Such things as air quality management are good use cases. You can't use your phone to do that. Why not? I am strictly against IoT in my household so I may be way off base, but why can't your phone control your air purifier?
mirker 4y ago
Privacy doesn’t matter unless the IoT devices are secure. Often times, they’re not.
hosh 4y ago
Adding an embedded LLM as a human interface for every appliance is a huge win— for consumers at least. For example, I have a dishwasher with a bunch of settings, can sense load, etc. It’s got a touch interface that works with wet hands. Or I can tell it to start with the usual settings, or that a particular load is a bit different. Same with the laundry, the pressure cooker. It is less mind bandwidth when you got kids. What I don’t want, is for my appliances to do is to phone home to the makers. LLMs (if you don’t somehow trigger its insanity) can be far more capable than Siri. How do you get that into something more energy efficient than a high end gaming rig? Something more hidden is using LLMs to reprogram machine-to-machine protocols. That might extend the lifetime of machines that have to talk with other machines, but it breaks planned obsolescence. There are plenty of exciting product ideas. Whether they are exciting revenue generators are another thing entirely.

brrrrrm 4y ago

ML is still in a compute hungry growth phase. Every exciting new achievement in accuracy requires more compute than before. It makes sense that tinyML, which has less compute, would always have lagging modeling support and be a bit less interesting. TinyML does result in much cheaper products, though.

cxie 4y ago

TinyML would depend on the breakthrough in Edge/embed AI chips, which doesn't seem to come in the near future...

jononor 4y ago
Why do you say that? There are microcontrollers announced with neural network accelerators integrated, and low power FPGAs with accelerators on the market, along with many chip startups aiming to provide milliwatt scale dedicated neural chips.

hosh 4y ago

That’s weird. Being able to use AI without being beholden to Big Tech seems like a huge win to me. Especially when I want to ensure that the AI is making best effort for my local community’s best interest, and not the best interest of a large corp.

mirker 4y ago
Even so, you can run models on a private cloud. No need to do anything but sense on an embedded device.
- hosh 4y ago
  That’s a huge burden on the natural resources (electricity). Beyond local control, I gave other examples of embedded AI in other comments — or more specifically, what you can do with an embedded LLM. Namely, being able to have a better human interface for complex settings, and being able to reprogram protocols (or anything that is “software-defined”) for future-proofing.
  - mirker 4y ago
    Integrating an accelerator into a dishwasher is also expensive. How many inferences are you going to get out of it before the hardware is outdated? Once a day for 5 years may not be enough to justify the cost. Compare that to having a single GPU in a server in your house that services all your IoT devices, which can be readily upgraded and requires no special software. The is no benefit to doing inference on device unless you are in the business of selling said devices.
    - hosh 4y ago
      A big part of TinyML is finding alternate paths for implementing AI in resource-constrained space. We already have SoC that is functionally not so much different from modern microcontrollers. Depending on economics of scale, it isn’t that big of a leap of imagination to see a microcontroller which includes 4-bit vector ops like a GPU.
  - 4y ago

RugnirViking 4y ago

From my work in AI & robotics, its generally finnicky and a bit crap even when a ton of work goes into it with a lot more compute (jetson nano or higher). I feel like trying to optimise those kinds of models is a bit niche/premature

steve_adams_86 4y ago
As a hobbyist that has been my perception of the situation as well, though I wasn’t sure if I was just missing something. The potential is immense and exciting, but try as I might, I can’t seem to make even fairly simple things work reliably as I want them to. One thing I love about embedded projects is that you can achieve pretty incredible reliability because everything can be so dialed in and isolated from points of failure in, say, an operating system. Trying to use ML for simple tasks felt like it eliminated that and introduced seemingly arbitrary failure into projects I really needed to work perfectly. Again, just a hobbyist, so I can’t make any broad statements. It seems to align with what you’re saying though. Once we get past this hump and have more powerful/effective models at a reasonable price point, I feel like it could be transformative. At the moment it’s still extremely interesting and fun to experiment with.

bambax 4y ago

I'm sure there are lots of fun things to do with an Arduino that could recognize gestures; I'm very surprised nobody's "excited" by this? For example, I made a photobooth simple device based on a Pi and a regular DSLR; the person takes a photo by pressing a button, then the Pi checks the camera and sends the latest image in a web gallery somewhere. The problem is the button: it needs a remote. If wired, it risks destroying the whole apparatus if someone pulls on the cord; if wireless, it risks being lost. A gesture-based trigger would be super cool; but having it run on the Pi through the DSLR risks damaging the camera, so it should run on a different device, cheap and not too power hungry, such as an Arduino. What about some kind of MIDI-controller based on an Arduino that could recognize gestures? I also made a webapp to learn sight-reading (babeloop.com) but it requires users clicking or tapping on the screen, which is not natural. It would be cool to be able to listen to users reading notes out loud and detect if they're right or wrong, on the fly, locally in a browser or on a phone. A light general speech recognition model such as VOSK is 40Mb and is able to recognize most phonemes; but for music sight reading, there are only 7 syllabes, so one should be able to make a much smaller model? I don't know how hard it would be though to train my own model...?

JohnFen 4y ago
> I'm sure there are lots of fun things to do with an Arduino that could recognize gestures Arduinos can recognize gestures, faces, etc. right now with well-established tech, no servers needed. I've had a couple of robots running around my place that do this sort of thing for a number of years now.
tiedieconderoga 4y ago
Embedded platforms generally have very little memory and very few hardware threads. Tiny SoCs will likely need accelerators to run more interesting models. For example, ARM is working on a ML coprocessor to pair with their Cortex-M chips. https://www.arm.com/products/silicon-ip-cpu/ethos/ethos-u55 https://www.arm.com/products/silicon-ip-cpu/ethos/ethos-u55

svantana 4y ago

I'm very into TinyML - not because of IoT but because small and fast models work well in (clientside) webpages, in realtime systems and pretty much everywhere. All while consuming less resources, which should be celebrated. Also, it's more democratic - anyone can compete in this field, supercomputer not needed. IMO, what is needed is a larger focus on resource consumption in competitions, benchmarks and rankings. For example, I like that you can rank some lists on paperswithcode by number of parameters [1]. [1] https://paperswithcode.com/sota/image-classification-on-imagenet?metric=Number%20of%20params&dimension=Top%201%20Accuracy https://paperswithcode.com/sota/image-classification-on-imag...

m3kw9 4y ago

Prob now you got GPT and there is a new baseline they have to live up to

Buttons840 4y ago

I've been learning how transformers work, and also reading about RNNs. It seems that transformers have succeeded because they can be parallelized, and solve many of the problems that RNNs have. This has lead to the advancements we've seen recently with GTP. Big companies are able to throw their data centers at the problem and train impressive and massive models. Part of me hopes that a non-parallelizable AI architecture might be discovered which performs even better. Perhaps the problems with RNNs could be solved in a way that doesn't parallelize? We would be so fortunate if a desktop computer could run an AI that's half as good as Microsoft's or Amazon's best AI. I would love to see the advantage of the data center removed. Philosophically, this does make some sense. The wisdom behind sayings such as "adding more people makes the project even later" exist because the greatest intellects we're aware of (ourselves) do not parallelize well.

bob1029 4y ago

After the initial obsession with Jupiter-scale models, we quickly discovered that very small models can perform even better if you reorganize your problem. Last week, we were trying to fine tune a 175b parameter model to take a natural language prompt with hopes of directly-outputting correct, domain-specific SQL. Now that reality has passed, we are looking at different paths. As of this week, we are trying to hit everything with the binary classification hammer. Turns out you can train a model to output 1 of 2 possible tokens with exponentially fewer parameters, training items, machine hours, etc. The statistics available in binary classification are also incredibly powerful and the results are trivial to reason with. Even if you need thousands of binary classifiers, their scale and granularity makes this a non-event or potentially an advantage. The real integration magic with AI/ML is starting to look like a weird form of set theory. At this level of complexity, detecting (and potentially confirming) the user's intention is way more important than trying to draw a direct map from input to destination.

tmtvl 4y ago

For anyone interested, it's not Meta Language ML (as in Standard ML), it's Machine Learning ML.