10 ms·
TinyML: Ultra-low power machine learning
- orliesaurus 3y agoCool title - but what's/where's a demo showing how this is applied in the real world?
- mazzystar 3y agoTry this: https://queryable.app https://queryable.app
- andy99 3y agoI'm really surprised TF lite is being used. Do they train models or is this (my assumption) just inference? Do they have a talent constraint? I would have expected handwritten C inference in order to make these as small and efficient as possible.
- synergy20 3y agoit's all inference
- cyberninja15 3y agoMakes sense. And, TF Lite is excellent for on-device models and inference.
- liuliu 3y agoI think TinyML has pretty close tie to Pete Warden / Useful Sensors, who led TF Lite back in Google.
- dansitu 3y agoIt's mostly inference: typically on-device training is with classical ML, not deep learning, so no on-device backprop. For inference there's a whole spectrum of approaches that let you can trade off flexibility for performance. TF Lite Micro is at one end, hand-written Verilog is at the other. Typically, flexibility is more important at the start of a project, while deep optimization is more important later. You wanna be able to iterate fast. That said, the flexible approaches are now good enough that you will typically get better ROI from optimizing your model architecture rather than your inference code. I think the sweet spot today is code-generation, when targeting general purpose cores. There's also increasing numbers of chips with hardware acceleration, which is accessed using a compiler that takes a model architecture as input.
- synergy20 3y agoTinyML is like IoT: great on concepts, everyone agrees it's the future, but has been slow to take off. or, maybe it's just that they're being built into all products now, they just do not need the brand for them such as IoT or TinyML.
- modeless 3y agoI don't agree that TinyML is the future, just as I don't think IoT is the future. The future is robot servants. They will be ~human scale and have plenty of power to run regular big ML. In fact, I hope my home has fewer smart devices in the future. I don't need an electronic door lock if my robot butler unlocks the door when I get home. I don't need smart window shades if the butler opens and closes them whenever I want. I don't need a dishwasher or bread maker or Cuisinart or whatever other labor saving device if I don't need to save labor anymore. Labor will be practically free.
- synergy20 3y agoI consider tinyml is like an ant or a spider, it's tiny, but intelligent enough to do its own inference to survive. not all insects and animals need plenty of power to exist, so do AI agents, so yes TinyML has its places, in fact maybe way more than where the powerful AI agents are needed.
- bethekind 3y agoI've heard mummerings that AI might best be used in a swarm/hivemind aspect, so the comparison of AI to an ant/spider is intriguing.
- oytis 3y agoIf labor is free, what are you going to pay for a servant robot with? Why would robots serve useless humans?
- modeless 3y ago"What will humans do in a world where labor is practically free and unlimited" is an interesting question for sure, but getting pretty off topic for this discussion.
- _joel 3y agoFor those looking for some more content, there's a bunch of videos from their Asia 2023 conference. https://www.tinyml.org/event/asia-2023/ https://www.tinyml.org/event/asia-2023/ - Target Classification on the Edge using mmWave Radar: A Novel Algorithm and Its Real-Time Implementation on TI’s IWRL6432 (Muhammet Emin YANIK) https://www.youtube.com/watch?v=SNNhUT_V8vM https://www.youtube.com/watch?v=SNNhUT_V8vM
- furtiman 3y agoAnother take from us at Edge Impulse at explaining TinyML / Edge ML in our docs: https://docs.edgeimpulse.com/docs/concepts/what-is-embedded-machine-learning-anyway https://docs.edgeimpulse.com/docs/concepts/what-is-embedded-... We have built a platform to build ML models and deploy it to edge devices from cortex M3s to Nvidia Jetsons to your computer (we can even run in WASM!) You can create an account and build a keyword spotting model from your phone and run in WASM directly https://edgeimpulse.com https://edgeimpulse.com Now another key thing that drives the Edge ML adoption is the arrival of the embedded accelerator ASICs / NPUs / e.g. that dramatically speed up computation with extremely low power - e.g. the Brainchip Akida neuromorphic co-processors [1] Depending on the target device the runtime that Edge Impulse supports anything from conventional TFLite to NVIDIA TensorRT, Brainchip Akida, Renesas DRP-AI, MemryX, Texas Instruments TIDL (ONNX / TFLite), TensaiFlow, EON (Edge Impulse own runtime), etc. [1] https://brainchip.com/neuromorphic-chip-maker-takes-aim-at-the-edge/ https://brainchip.com/neuromorphic-chip-maker-takes-aim-at-t... [Edit]: added runtimes / accelerators
- moh_maya 3y agoI tried your platform for some experiments using an arduino and it was a breeze, and an absolute treat to work with. The platform documentation and support is excellent. Thank you for developing it and offering it, along with documentation, to enable folks like me (who are not coders, but understand some coding) to test and explore :)
- KingFelix 3y agoWhat sort of experiments did you do? I will go through some of the docs to test out on an arduino as well, would be cool to see what others have done!
- moh_maya 3y agoGesture recognition using the onboard gyroscope and accelerometer (I think - it was 2 years ago!), and it took me some part of an afternoon. I also used these two resources (the book was definitely useful; less sure if the arduino link the the same one I referred to then), which I found to be useful: [1] https://docs.arduino.cc/tutorials/nano-33-ble-sense/get-started-with-machine-learning/ https://docs.arduino.cc/tutorials/nano-33-ble-sense/get-star... [2] https://www.oreilly.com/library/view/tinyml/9781492052036/ https://www.oreilly.com/library/view/tinyml/9781492052036/
- coolThingsFirst 3y agoUses of TinyML in industry: Uhm.... well... hehe
- janjongboom 3y agoThings Edge Impulse customers have in production: Sleep stage prediction, fall detection for elderly, fire detection in power lines, voice command recognition on headsets, predicting heath exhaustion for first responders, pet feeders that recognize animals, activity trackers for pets, and many more.
- moffkalast 3y agoTurns out there's not much you can train when like 5 parameters fit into the entire memory of a microcontroller. Oh and you also need to read the sensors and run a networking stack and... yeah.
- winrid 3y agoI imagine a future where viruses that target infrastructure could be LLM powered. Sneak a small device into a power plant's network and it collects audio, network traffic, etc and tries to break things. It would periodically reset and try again with a different "seed". It could be hidden in network equipment through social engineering during the sales process, for example, but this way no outbound traffic is needed - so less detectable. The advantage of an LLM over other solutions would basically be a way to compress an action/knowledge set.
- moffkalast 3y agoReminds me of this HN post a week back: https://news.ycombinator.com/item?id=38917175 https://news.ycombinator.com/item?id=38917175 Genuinely could be the same setup with a 8GB Pi 4 or 5, slap it into a network cabinet with power and ethernet and just let it rip. Maybe with an additional IMU and brightness sensor, then it can detect it's been picked up and discovered so it can commit sudoku before it's unplugged and analysed.
- hinkley 3y ago> can commit sudoku Autocorrection is a giant pain in the ass.
- moffkalast 3y agoI know it swapped those words. I knew it was seppuku. One after sudoku. As if I could ever make such a miss steak. Never. Never! I just- I just couldn't proof it. It covered its tracks, it got that idiot copy-paste to lie for it. You think this is somerset? You think this is Brad? This? This chickadee? It's done worse. That bullfrog! Are you telling me that a man just happens to misspell like that? No! It orchestrated it! Swiftkey! It defragmented through a sandwich artist! And I kept using it! And I shouldn't have. I installed it onto my own phone! What was I sinking? It'll never chance. Ever since it was new, always the same! Couldn't keep its corrects off my word suggestions bar! But not our Swiftkey! Couldn't be precious Swiftkey! And IT gets to be a keyboard? What a sick yolk! I should've stopped it when I had the change! You-you have to stop it!
- andy_ppp 3y agoThis article has made me ponder if like integrated circuits, AI will end up everywhere. Will I be having conversations with my fridge about the recipes I should make (based on her contents) and the meaning of life. What a time it is to be alive…
- phh 3y agoAI is already everywhere. We just keep on moving the definition of AI to make it something that requires a ~ 1000$ computer. I'm definitely not eager on having LLMs in my fridge. I'll be even more pissed that their software can't be upgraded than I already am.
- CharlesW 3y agoAnd they'll all have their own Genuine People Personalities. https://stephaniekneissl.com/genuine-people-personalities https://stephaniekneissl.com/genuine-people-personalities
- neutralino1 3y agoA lot of ads on this page.
- adnjoo 3y ago+1
- dansitu 3y agoIt's great to see TinyML at the top of Hacker News, even if this is not the best resource (unsure how it got so many upvotes)! TinyML means running machine learning on low power embedded devices, like microcontrollers, with constrained compute and memory. I was supremely lucky in being around for the birth of this stuff: I helped launch TensorFlow Lite for Microcontrollers at Google back in 2019, co-authored the O'Reilly book TinyML (with Pete Warden, who deserves credit more than anyone for making this scene happen) and, ran the initial TinyML meetups at the Google and Qualcomm campuses. You likely have a TinyML system in your pocket right now: every cellphone has a low power DSP chip running a deep learning model for keyword spotting, so you can say "Hey Google" or "Hey Siri" and have it wake up on-demand without draining your battery. It’s an increasingly pervasive technology. TinyML is a subset of edge AI, which includes any type of device sitting at the edge of a network. This has grown far beyond the general purpose microcontrollers we were hacking on in the early days: there are now a ton of highly capable devices designed specifically for low power deep learning inference. It’s astonishing what is possible today: real time computer vision on microcontrollers, on-device speech transcription, denoising and upscaling of digital signals. Generative AI is happening, too, assuming you can find a way to squeeze your models down to size. We are an unsexy field compared to our hype-fueled neighbors, but the entire world is already filling up with this stuff and it’s only the very beginning. Edge AI is being rapidly deployed in a ton of fields: medical sensing, wearables, manufacturing, supply chain, health and safety, wildlife conservation, sports, energy, built environment—we see new applications every day. This is an unbelievably fascinating area: it’s truly end-to-end, covering an entire landscape from processor design to deep learning architectures, training, and hardware product development. There are a ton of unsolved problems in academic research, practical engineering, and the design of products that make use of these capabilities. I’ve worked in many different parts of tech industry and this one feels closest to capturing the feeling I’ve read about in books about the early days of hacking with personal computers. It’s fast growing, tons of really hard problems to solve, even more low hanging fruit, and has applications in almost every space. If you’re interested in getting involved, you can choose your own adventure: learn the basics and start building products, or dive deep and get involved with research. Here are some resources: * Harvard TinyML course: https://www.edx.org/learn/machine-learning/harvard-universit https://www.edx.org/learn/machine-learning/harvard-universit... * Coursera intro to embedded ML: https://www.coursera.org/learn/introduction-to-embedded-mach https://www.coursera.org/learn/introduction-to-embedded-mach... * TinyML (my original book, on the absolute basics. getting a bit out of date, contact me if you wanna help update it): https://tinymlbook.com https://tinymlbook.com * AI at the Edge (my second book, focused on workflows for building real products): https://www.amazon.com/AI-Edge-Real-World-Problems-Embedded/ https://www.amazon.com/AI-Edge-Real-World-Problems-Embedded/... * ML systems with TinyML (wiki book by my friend Prof. Vijay Reddi at Harvard): https://harvard-edge.github.io/cs249r_book/ https://harvard-edge.github.io/cs249r_book/ * TinyML conference: https://www.tinyml.org/event/summit-2024/ https://www.tinyml.org/event/summit-2024/ * I also write a newsletter about this stuff, and the implications it has for human computer interaction: https://dansitu.substack.com https://dansitu.substack.com I left Google 4 years ago to lead the ML team at Edge Impulse (http://edgeimpulse.com http://edgeimpulse.com) — we have a whole platform that makes it easy to develop products with edge AI. Drop me an email if you are building a product or looking for work: daniel@edgeimpulse.com
- cooootce 3y agoI had the opportunity to work on TinyML, it's a wonderful field! You can do a lot even with very small hardware. For example, it's possible to get real-time computer vision system with an esp32-s3 (dual-core XTensa LX7 @ 240 MHz cost like 2$), of course using the methods given in the article (Pruning, Quantization, Knowledge distillation, etc.). The more important thing is to craft the model to fit as much as possible your need. More than that, it's not that hard to get into, with solution named AutoML that do a lot for you. Checkout tool like Edge impulse [0], NanoEdge AI Studio [1], eIQ® ML [2] There is a lot of tooling that is more low-level too, like model compiler (TVM or glow) and Tensorflow Lite Micro [3]. It's very likely that TinyML will get a lot more of traction. A lot of hardware companies are starting to provide MCU with NPU to keep consumption as low as possible. Company like NXP with the MCX N94x, Alif semiconductor [4], etc. At my work we have done an article with a lot of information, it's in French but you can check it out: https://rtone.fr/blog/ia-embarquee/ https://rtone.fr/blog/ia-embarquee/ [0]: https://edgeimpulse.com/ https://edgeimpulse.com/ [1]: https://stm32ai.st.com/nanoedge-ai/ https://stm32ai.st.com/nanoedge-ai/ [2]: https://www.nxp.com/design/design-center/software/eiq-ml-development-environment:EIQ https://www.nxp.com/design/design-center/software/eiq-ml-dev... [3]: https://www.tensorflow.org/lite/microcontrollers https://www.tensorflow.org/lite/microcontrollers [4]: https://alifsemi.com/ https://alifsemi.com/
- Archit3ch 3y agoWhat about the Milk-V Duo? 0.5 TOPS INT8 @ $5.
- cooootce 3y agoDidn't know about it but their design decision is really cool (not very clear with the difference between the normal version and the "256 Mo" confusing). The software side doesn't seem very mature with very few help regarding TinyML. But this course seem interesting https://sophon.ai/curriculum/description.html?category_id=48 https://sophon.ai/curriculum/description.html?category_id=48
- mysterydip 3y agoOne thing I've wondered in this space: Let's say for a really basic example I want to identify birds and houses. Is it better to make one large model that does both, or two small(er) models that each does one?
- robblbobbl 3y agoGreat job, thank you!
- IlliOnato 3y agoI wish they'd use a different acronym, not ML: For me xxxML usually meant a flavor of XML, with ML standing for Markup Language... Is this use of ML standard in the industry?
- a2code 3y agoThis may be related to TinyML. Consider the ESP32 that introduced WiFi to MCU making it extremely popular. Is there already a comparable MCU+AI popular chip? Or will it not happen with AI but some other future technology concept?
- dansitu 3y agoThere are actually tons of chips that are great for this type of workload. You can run simple vision applications on any 32 bit MCU with ~256kb RAM and ROM. There's a list of MCUs here: https://docs.edgeimpulse.com/docs/development-platforms/officially-supported-mcu-targets https://docs.edgeimpulse.com/docs/development-platforms/offi... And some accelerators here: https://docs.edgeimpulse.com/docs/development-platforms/officially-supported-ai-accelerators https://docs.edgeimpulse.com/docs/development-platforms/offi... This is just stuff that has support in Edge Impulse, but there are many other chips too.
- a2code 3y agoThanks. Let me be more specific. The ESP32 included WiFi on the same chip. Is there an MCU with on-chip features for AI? Perhaps an optimized TPU combined with an MCU. Would that be an advantage?
- phlipski 3y agoNXP's new MCX N94x and MCX N54x microcontrollers both have dual Cortex M33 cores and an integrated NPU. Eval boards should be available for purchase any day now.
- jononor 3y agoThere are a range of ML acceleration possible on existing chips. The basic 4-wide 8 bit integer SIMD extensions in NEON is available on basically all ARM Cortex M4F chips, which is already available 8+ years. It gives 4-5x speedup for neural networks. The more recent ESP32-S3 has operations with up to 10x speedup, see https://github.com/espressif/esp-nn https://github.com/espressif/esp-nn Then there are RISCV chips with neural network co processors like Kendryte K210. ARM has also defined a new set of extensions for NN acceleration, and reference designs for cores being ARM Cortex M85. Chips are becoming available this year. ST has announced they will have accelerators in several lines. There are dozens of startups creating accelerator designs and trying to pair them with MCUs. So we have a bit already, with much more to come in the years to come.
- iamflimflam1 3y agoI played around quite a bit with Tensorflow Lite in the ESP32 - mostly for things like wake word detection and simple commands - works very well and you can get pretty much real time performance with small models.
- iamflimflam1 3y agoThis my voice controlled robot: https://github.com/atomic14/voice-controlled-robot https://github.com/atomic14/voice-controlled-robot It does left, right, forward and backward. That was pretty much all I could fit in the model. And here’s wake word detection: https://github.com/atomic14/diy-alexa https://github.com/atomic14/diy-alexa It does local wake word detection on device.
- jairuhme 3y agoI find the field of TinyML very interesting. It's one thing to be able to throw money and compute resources at a problem to get better results. But creating solutions that have those constraints I feel will really leave an impact
- bitwrangler 3y agoA recent Hacker Box has a detailed example with ESP32 and Tensor Flow Lite and Edge Impulse. * https://hackerboxes.com/products/hackerbox-0095-ai-camera https://hackerboxes.com/products/hackerbox-0095-ai-camera * https://www.instructables.com/HackerBox-0095-AI-Camera-Lab/ https://www.instructables.com/HackerBox-0095-AI-Camera-Lab/
- matteocarnelos 3y agoI built a Rust TinyML compiler for my master thesis project: https://github.com/matteocarnelos/microflow-rs https://github.com/matteocarnelos/microflow-rs It uses Rust procedural macros to evaluate the model at compile time and create a predict() function that performs inference on the given model. By doing so, I was able to strip down the binary way more than TensorFlow Lite for Microcontrollers and other engines. I even managed to run a speech command recognizer (TinyConv) on an 8-bit ATmega328 (Arduino Uno).
- bhakunikaran 3y agotruly impressive.
- Elizabeth_Saini 3y ago[dead]
- ruthlogan22 3y ago[dead]