6 ms·
YOLOv5 on FPGA with Hailo-8 and 4 Pi Cameras
- tonetegeatinst 2y agoOpencv, YOLO, and darknet were some of the best tools back when I was doing research projects with this. While it is way more efficient compared to say feeding it into a RAG LLM.....it is process intensive. Its hard to handle multiple data streams of video, and I think I maxed out around 20 camera feeds before the computer slowed to a crawl. NVME storage and better internet would defiantly push the limit on what is possible, but unfortunately it is impossible to get good internet where I live. Also, cameras are not cheap and neither is flash storage. But I do know this stuff sees use in both commercial and I bet they use some better version for the defense industry.
- zimpenfish 2y agoThat looked interesting as a self-hostable project until it got to the requirement for a $3200 AMD board. Maybe the price will come down one day...
- 15155 2y ago~400kLUT is very obtainable these days - if you don't need the ARM cores: https://www.lcsc.com/product-detail/Programmable-Logic-Device-CPLDs-FPGAs_AMD-XILINX-XC7K410T-2FFG900I_C2937666.html https://www.lcsc.com/product-detail/Programmable-Logic-Devic... (420T/480T are priced similarly new.)
- DoingIsLearning 2y agoThe ultrascale is definitely pricy for non-industry applications. The FPGA design is probably larger because of the 4x camera pipeline. Perhaps with a single camera you could port this to fit on a Zynq 7000 footprint with something like a Pynq Z1 or Numato Styx, which are in the $250 hobbies price point for example.
- 15155 2y agoNo chance does the Zynq 701x series perform capably for this task: it's weak Artix fabric and there's only ~50kLUT on the common parts. Even fully pipelined designs, especially those that interface with the hard AXI bus, will struggle to clock above 50MHz. The Zynq Z7030+ chip would probably be able to get the job done but aren't as common - https://www.lcsc.com/product-detail/Programmable-Logic-Device-CPLDs-FPGAs_AMD-XILINX-XC7Z030-2FFG676I_C966962.html https://www.lcsc.com/product-detail/Programmable-Logic-Devic.... The Kintex 410T-480T are available new <$160 from reputable vendors, used <$50 from less-reputable vendors - they do have the performance (and overall IO) for this task.
- dailykoder 2y agoAt the moment I am wondering if I could build an accelerator for george hotz' tinygrad[1] with cheap FPGAs (i do have an arty a7 35K, this might be too small, i guess?). According to the readme it shall be "easy" to add new accelerator hardware. Sadly my knowledge is still a bit limited around all the python-machinelearning-ecosystem, but if I understand it correctly you "just" need an openCL kernel and need to be able to shove the data back and forth somehow. Didn't have enough time to dive into it yet and still working on some other project, but this still tickles the back of my head and would be cool even if I could only run mnist on it. - [1] https://github.com/tinygrad/tinygrad/ https://github.com/tinygrad/tinygrad/
- gh02t 2y ago> but if I understand it correctly you "just" need an openCL kernel and need to be able to shove the data back and forth somehow. To use it with an FPGA accelerator you also have to build all the "hardware" to run said openCL kernel efficiently, manage data transfer, talk to the host, etc for the FPGA. This is very foreign if you're only used to doing software design and still very nontrivial even if you've done FPGA work, though I think there are some open hardware projects around doing this.
- dailykoder 2y agoYeah, i think the, especially efficient, data transfer would be my biggest enemy. I did quite a lot FPGA stuff in recent years, mainly CPU design stuff, a little pipelining and some hardware-software-codesign. So I should just see it as a toy project, start with said mnist to get atleast something running, don't matter if it's efficient and then work my way up. For example I haven't done anything PCIe related yet, but I guess there is enough IP available
- 15155 2y agoTransceivers are the only option for high-speed connectivity - this means PCIE or 100GbE.
- sorenjan 2y agoDepending on what you're trying to do you might not need it. A popular option for object detection people use with their home hosted video surveillance (using Frigate and Home Assistant) is the Coral TPU and Raspberry Pi or some decommissioned thin client.
- muxamilian 2y agoGoogle Coral seems abandoned by Google. Not official but last news on their page from May 2022
- geerlingguy 2y agoI've heard from the folks at Pineboard[1] there's been some new activity, as this year manufacturing ramped back up... it's still decent hardware but is getting long in the tooth. Regarding the Hailo featured in the article, a few of us have been messing with it on a Raspberry Pi 5[2], and it offers more performance in a similar power envelope. The major downside is availability. I can buy the Coral on many electronics supplier sites, but Hailo seems to be selling through 'Product inquiry' right now, which is not easy to navigate as an individual! [1] https://pineboards.io https://pineboards.io [2] https://pipci.jeffgeerling.com/cards_m2/hailo-8-ai-module.html https://pipci.jeffgeerling.com/cards_m2/hailo-8-ai-module.ht...
- algo_trader 2y agoAny good write up on Hailo and similar ? BTW, is it possible to have 10 of these connected to a single board/cpu ?
- muxamilian 2y agoMaybe not 10 but in principle it’s possible: https://www.lannerinc.com/products/edge-ai-appliance/deep-learning-accelerators/falcon-h8 https://www.lannerinc.com/products/edge-ai-appliance/deep-le...
- geerlingguy 2y ago
- scottapotamas 2y agoGreat writeup. Always a treat to see more high quality FPGA project postmortems, even if they aren’t using accessible parts/toolchains.
- daghamm 2y agoI am not familiar with this NN accelerator. Does anyone have a comparison between Hailo and, say, a mid or high-end GPU or a TPU?
- michaelt 2y agoAccording to [1] the manufacturers claim 2% of the TOPS of a RTX 4090, and only 0.8% the power consumption. $200 in prototype quantities [2] which is 12% the price of a 4090 - but perhaps the price drops when you order them in bulk? They claim it compares favourably to an 'Nvidia Xavier NX' for image classification tasks, providing somewhat more FPS at significantly lower power consumption. 218 fps running YOLOv5m on 640×640 inputs. They're completely silent about the amount of memory it has, but you can fit int8 YOLOv5m into about 20 MB so it'll certainly be an amount measured in megabytes rather than gigabytes. Their target market is "CCTV camera that tracks cars and people" rather than "Run an LLM" or "train a network from scratch" [1] https://hailo.ai/products/ai-accelerators/hailo-8-ai-accelerator/#hailo8-performance https://hailo.ai/products/ai-accelerators/hailo-8-ai-acceler... [2] https://up-shop.org/hailo-m2-key.html https://up-shop.org/hailo-m2-key.html
- muxamilian 2y agoFrom what I know it doesn’t have memory but streams everything to the chip. So there’s no limit regarding the size of the the neural network (unlike Google Coral)
- michaelt 2y agoMaybe? If a 20 MB network achieves 218 fps that'd need 4.3 GB/s of bandwidth just to stream the network, completely ignoring the images. And they use PCIe Gen3 which is 1 GB/s per lane, with different products having 1, 2 or 4 lanes. So... maybe just about?
- geerlingguy 2y agoSome of their marketing mentions no need for external DRAM, for example, from this CNX article[1]: "One of the key reasons for the performance improvement is that RAM is self-contained without the need for external DRAM like other solutions. This decreases latency a lot and reduces power consumption." Not sure how much RAM is included on the chip, but I'm also thinking in the tens of MB range, certainly not gigabytes. [1] https://www.cnx-software.com/2020/10/07/learn-more-about-hailo-8-ai-accelerator-and-understanding-ai-benchmarks/ https://www.cnx-software.com/2020/10/07/learn-more-about-hai...
- globalnode 2y agoSome of the people didnt look too happy about you filming them, then you went and put them online for the world to see, classy.
- bigyikes 2y agoBreaking: people in public don’t want to be seen by the public. It’s a detailed breakdown of a technically impressive project and your main takeaway is the 5 seconds in the demo where a guy covers his face? Kudos to the author for making something neat and sharing it.
- shaky-carrousel 2y agoBreaking: people in public don't want to be recorded and uploaded on the internet for millions to see. Crazy, right?
- Workaccount2 2y agoWell yeah, it is kind of crazy. Any scenario in which undo harm is brought upon someone because they were a passerby on the street in a video, is a such a reach that you have to question how grounded in reality they are. It's some deep "I'm the main character" level thinking.
- IncreasePosts 2y agoOh, the horror. Holy shit was that Jason walking by? He told me he was in Alaska...I'm going to need to have a talk with him. Thank goodness I ran across this video.
- yazzku 2y agoA good point that shouldn't be dismissed so quickly. This tracking also has little use beyond surveillance, so one has to wonder what is it that the author thinks they are doing, or why they think it's interesting or useful. That they then go ahead and film a crowd without their consent is more telling of their position on this moral question than it is any direct harm to the people in the video.
- shrubble 2y agoWhat is the ultimate delivery after all this work? Did it correlate/track the same people across multiple video feeds, for instance?
- gte525u 2y agoIt's line speed processing of multiple cameras in HW - it should be less power consumption than equivalent GPU or Jetson.
- throwaway2562 2y agoThis makes me think of monkeys cheerfully building their own cage. Sorry, but I can’t just ‘cool-project-bro’ this one. Does nobody else have the faintest misgivings about where we’re at right now: human surveillance as just scratching a technical itch? Apologies if this comes over all grumpy, but wow. Seriously.
- _giorgio_ 2y agoWhat's a good (production ready) setup? I'm thinking of an external camera (weather resistant) and hardware. The hardware could be a small computer that connects to the camera (maybe with wifi?), and runs the YOLO model.
- brk 2y agoThat already exists. Lilin is one of several CCTV companies implementing on-camera YOLO. Axis, Hanwha, and i-Pro all have options for you to run your own software/models on camera as well.
- _giorgio_ 2y agoOk, thanks. I'll look into that. I hope that they don't require some specific abstruse format for the models!
- eurekin 2y agoAfter years of wondering, I have to ask. What are real life actual useful cases for this tech? I can imagine in manufacturing: detecting defects or layout mismatch - that's one. Is there any open source project that uses a image recognition library to achieve any useful task? All I've seen from board partners seem to at most provide very simple demos, where a box with label is drawn around an object. Who actually is using that information, how and for what? I've also been a part of the Kinect craze and made 3 demos (games mostly) using their SDK and still have a very hard time defending this tech in eyes of coworkers that only see this as a surveillance tech
- mmcwilliams 2y agoI've used YOLOv5 models in robotics for object detection. While VLMs are great at describing images more generally, having a bounding box of a detection with a confidence score is very useful when paired with depth cameras for locating objects in an open environment. Especially when it can be run on-board and at framerate.
- yazzku 2y agoTo surveil people in the streets.
- mechagodzilla 2y agoAs you guessed, high-speed machine vision stuff is frequently used in manufacturing settings for sorting or various quality control tasks. Imagine a picking out bad potatoes on a conveyor belt moving 10s of potatoes per second, or identifying particle counts and size distributions in a stream of water to gauge water quality.