14 ms·
SIMD-accelerated computer vision on a $2 microcontroller
- restricted_ptr 2y agoI wonder if ESP32 has VLIW slots and a tighter instruction packaging is possible?
- duskwuff 2y agoNeither Xtensa nor RISC-V are VLIW architectures.
- restricted_ptr 2y agoXtensa architecture is flexible and extendable by the user. Ability to define new instructions, hw features and VLIW configurations are some of the key features. You can find more details on the internet https://en.m.wikipedia.org/wiki/Tensilica https://en.m.wikipedia.org/wiki/Tensilica
- jki275 2y agoI don't think that applies to the ESP32 family of devices. I've never heard of DSP hardware onboard them. I think the comment you're referring to is talking about the architecture in general, but not the silicon we're discussing here.
- restricted_ptr 2y agoESP32 ee.* operations in assembly look pretty much like aliases for a VLIW bundles, on the same cycle issuing loads used in the next op while also doing multiplication on other operands. This is not a minimal Xtensa. They might not have the Tensilica toolchain for redistribution to use these features freely but apparently they exposed these extensions in their assembler in some form.
- thrtythreeforty 2y agoGenerally speaking, this is not correct. Base Xtensa is not VLIW, but Xtensa's various vector extensions do allow VLIW instructions, collectively called "FLIX." It is doubtful that ESP32's Xtensa is VLIW-capable, though. Presumably their compiler would emit FLIX instructions if it were.
- westurner 2y ago> As I've been really interested in computer vision lately, I decided on writing a SIMD-accelerated implementation of the FAST feature detector for the ESP32-S3 [...] > In the end, I was able to improve the throughput of the FAST feature detector by about 220%, from 5.1MP/s to 11.2MP/s in my testing. This is well within the acceptable range of performance for realtime computer vision tasks, enabling the ESP32-S3 to easily process a 30fps VGA stream. What are some use cases for FAST? Features from accelerated segment test: https://en.wikipedia.org/wiki/Features_from_accelerated_segment_test https://en.wikipedia.org/wiki/Features_from_accelerated_segm... Is there TPU-like functionality in anything in this price range of chips yet? Neon is an optional SIMD instruction set extension for ARMv7 and ARMv8; so Pi Zero and larger have SIMD extensions Orrin Nano have 40 TOPS, which is sufficient for Copilot+ AFAIU. "A PCIe Coral TPU Finally Works on Raspberry Pi 5" https://news.ycombinator.com/item?id=38310063 https://news.ycombinator.com/item?id=38310063 From https://phys.org/news/2024-06-infrared-visible-device-2d-material.amp https://phys.org/news/2024-06-infrared-visible-device-2d-mat... : > Using this method, they were able to up-convert infrared light of wavelength around 1550 nm to 622 nm visible light. The output light wave can be detected using traditional silicon-based cameras. > "This process is coherent—the properties of the input beam are preserved at the output. This means that if one imprints a particular pattern in the input infrared frequency, it automatically gets transferred to the new output frequency," explains Varun Raghunathan, Associate Professor in the Department of Electrical Communication Engineering (ECE) and corresponding author of the study published in Laser & Photonics Reviews. "Show HN: PicoVGA Library – VGA/TV Display on Raspberry Pi Pico" https://news.ycombinator.com/item?id=35117847#35120403 https://news.ycombinator.com/item?id=35117847#35120403 https://news.ycombinator.com/item?id=40275530 https://news.ycombinator.com/item?id=40275530 "Designing a SIMD Algorithm from Scratch" https://news.ycombinator.com/item?id=38450374 https://news.ycombinator.com/item?id=38450374
- shraiwi 2y agoThanks for reading! > What are some use cases for FAST? The FAST feature detector is an algorithm for finding regions of an image that are visually distinctive, which can be used as a first step in motion tracking and SLAM (simultaneous localization and mapping) algorithms typically seen in XR, robotics, etc. > Is there TPU-like functionality in anything in this price range of chips yet? I think that in the case of the ESP32-S3, its SIMD instructions are designed to accelerate the inference of quantized AI models (see: https://github.com/espressif/esp-dl https://github.com/espressif/esp-dl), and also some signal processing like FFTs. I guess you could call the SIMD instructions TPU-like, in the sense that the chip has specific instructions that facilitates ML inference (EE.VRELU.Sx performs the ReLU operation). Using these instructions will still take away CPU time where TPUs are typically their own processing core, operating asynchronously. I’d say this is closer to ARM NEON.
- evanjrowley 2y agoA comparable board is the ESP32-CAM, which is supported by this really practical computer vision project: https://github.com/jomjol/AI-on-the-edge-device?tab=readme-ov-file https://github.com/jomjol/AI-on-the-edge-device?tab=readme-o...
- maven29 2y agoThere is an ESP32-S3 version of this camera breakout board, which is presumably what OP might have used for prototyping. The S3 variant easily justifies the slight additional cost, given that it's easily faster by an order of magnitude or greater, having SIMD and an FPU. https://github.com/espressif/esp-dl/tree/master/examples/face_recognition https://github.com/espressif/esp-dl/tree/master/examples/fac...
- hi-v-rocknroll 2y agoIn the CV department, I recently ordered a cheap FPGA + ARM Cortex-M3 + 64 Mbit SRAM + 32 Mbit flash that does camera input and HDMI output. Like a budget Zynq for CV. https://wiki.sipeed.com/hardware/en/tang/Tang-Nano-4K/Nano-4K.html https://wiki.sipeed.com/hardware/en/tang/Tang-Nano-4K/Nano-4... https://www.aliexpress.us/item/3256806880637138.html https://www.aliexpress.us/item/3256806880637138.html
- unwind 2y agoCool board! Would any of the "retro" game/home computer firmwares fit in that FPGA? I find comparing capacity hard for stuff like that.
- londons_explore 2y agoYes, easily, but unless someone has done it already, 'porting' them to this board would be a lot of work.
- hi-v-rocknroll 2y agoThere's absolutely no reason ROMs have to waste scarce resources of a hybrid FPGA. Micro SD cards (called TF in China) and eMMC are the usual solutions. Example: https://www.aliexpress.us/item/3256806498688867.html https://www.aliexpress.us/item/3256806498688867.html
- picture 2y agoAlso see this short post about SIMD on ESP32-S3, discussed previously. https://bitbanksoftware.blogspot.com/2024/01/surprise-esp32-s3-has-few-simd.html https://bitbanksoftware.blogspot.com/2024/01/surprise-esp32-...
- rurban 2y agoWe prefer something more expensive and better: https://up-board.org/upsquared/specifications/ https://up-board.org/upsquared/specifications/ Intel UpSquared
- rowanG077 2y agoMore expensive sure. But better is pretty rich considering it is Intel. My money is on this platform just evaporating in the next 5 years. Esp32 has proven you can rely on supply and longevity.
- c0balt 2y agoArguably the UP^2 is another class of device. Up to 8 GB of RAM and up to 128 GB of storage + a whole x86 CPU with dual gigabit LAN. And the price, size and power consumption are also quite a bit higher but it will certainly grant a better general compute environment, if you want to run Linux or smth.
- rurban 2y agoIt's very easy to use any pytorch or tensorflow packages, or open3d, pcl, librealsense or similar vision packages. Powerful enough to do realtime vision tasks, which you certainly cannot do with 2€ boards.
- DeathArrow 2y ago>For silicon that's cheaper than the average coffee, that's pretty cool. Maybe it's not the chip that it's too cheap. Maybe it's the coffee that's too expensive.
- deleted 2y ago[deleted]
- mppm 2y agoOTOH, I've been waiting for disposable coffee cups with OLED-based video ads ever since Minority Report. But tech progress is just too damn slow :P
- yjftsjthsd-h 2y agoI dunno about OLED, but now that you say it the costs do make some sort of "smart" coffee disturbingly plausible.
- TheAdamist 2y agoBased on the recent post about the disposable Montreal subway tickets with a super cheap nfc chip (and amusingly on a paper ticket with a a printed on fake smart chip connection) it should be super cheap to have an automated kiosk that pairs your drink order to a paper cup that when a barista swipes shows your cup your order shows up or fills it automatically. https://www.righto.com/2024/06/montreal-mifare-ultralight-nfc.html?m=1 https://www.righto.com/2024/06/montreal-mifare-ultralight-nf... (It was linked from here but i don't have the HN link)
- BizarroLand 2y agoSeems like it would be cheaper to have a simple QR code be thermally printed on the cup. A fraction of a penny's worth of chemical spritz on the bottom of the cup would do the trick.
- surfingdino 2y ago
- ladyanita22 2y agoAnyone with experience on Rust for ESP32 controllers could chime in on whether this is feasible on rust as well?
- f_devd 2y agoIt is possible, mainly depends on LLVM/clang support as rust ASM is very easy to do
- Qwuke 2y agoCompared to ESP8266, there's generally pretty good ESP32 support for Rust, but you'll likely need to use in your C++ toolchain if you want to use the standard library. no-std in Rust for ESP32 isn't terrible in my experience, though, just not as fleshed out - particularly for hooking into components like wifi/networking and probably a camera as well. Like the other commenter said, there's plenty of support for SIMD and asm in Rust. You might ask around on a Rust embedded or Rust ESP32 chatroom before making the dive.
- the__alchemist 2y agoYou can actually use the IDF system in Rust to use the std lib, at least on ESP32-C3. Probably others too. If you are on Windows, you will need to place the project folder at the top level drive directory, and there are other quirks as well, but it works.
- Qwuke 2y agoHaha yes, I misphrased a bit but that's what I meant when I said you'll need to use C++ to use the stdlib. It's not quite pure embedded Rust but yes it does work.
- sylware 2y agoYep, SIMD seems to win the race vs SMT for that type of processing.
- hajile 2y agoI don't think SIMD and SMT is an either/or proposition. SMT-4 or SMT-8 with a bunch of SIMD has the potential to get better perf/area due to the threads hiding the latency.
- dansitu 2y agoIf you're interested in this stuff and wanna try it yourself, check out our product, Edge Impulse: https://edgeimpulse.com/ai-practitioners https://edgeimpulse.com/ai-practitioners We work directly with vendors to perform low level optimization of deep learning, computer vision, and DSP workloads for dozens of architectures of microcontrollers and CPUs, plus exotic accelerators (neuromorphic compute!) and edge GPUs. This includes ESP32: https://docs.edgeimpulse.com/docs/edge-ai-hardware/mcu/espressif-esp32 https://docs.edgeimpulse.com/docs/edge-ai-hardware/mcu/espre... You can upload a TensorFlow, PyTorch, or JAX model and receive an optimized C++ library direct from your notebook in a couple lines of Python. It's honestly pretty amazing. And we also have a full Studio for training models, including architectures we've designed specifically to run well on various embedded hardware, plus hardware-aware hyperparameter optimization that will find the best model to fit your target device (in terms of latency and memory use).
- TheMagicHorsey 2y agoYo! This is awesome stuff!
- dansitu 2y agoThank you! We're trying to bring embedded ML in reach of all engineering teams and domain experts. Previously you needed a crazy mixture of ML knowledge and low-level embedded engineering skills even to get started, which is not a common occurrence!
- RobotToaster 2y agoI don't think the output from this can be used in any open source project due to the community plan restrictions, FYI.
- dansitu 2y agoThat's definitely not our intention: the output of all Community projects is by default Apache 2.0 licensed, unless the developer specifies a different one. The community plan does have commercial use restrictions; it's designed for education, demos, and research. We have a pretty good presence in the academic community with tons of papers, code, and projects developed using our community version. Here's a Google Scholar search showing a bunch of papers: https://scholar.google.com/scholar?start=0&q=%22edge+impulse%22&hl=en&as_sdt=0,41 https://scholar.google.com/scholar?start=0&q=%22edge+impulse... We also have our own public sharing platform: https://edgeimpulse.com/projects/overview https://edgeimpulse.com/projects/overview
- robxorb 2y agoI wonder how hard it would be, presumably with some trade-off with detection windows, to use a few of these in parallel and process higher resolutions and frame rates?
- rldjbpin 2y agotinyml fascinates me because its principles can be directly applied to web-based applications imho. micropython seems pretty accessible from first glance. would it be easy to create a webassembly port of its code?
- jononor 2y agoMicroPython port for WASM/JS already exists. For device usecases, one can play with it at https://micropython.org/unicorn/ https://micropython.org/unicorn/ It can be used in PyScript for client side development. And has JavaScript/DOM bridge as well. https://pyscript.net/tech-preview/micropython/about.html https://pyscript.net/tech-preview/micropython/about.html
- hoseja 2y agoAm I reading wrong or is the penultimate part just basic two's complement?