3 ms·

This looks interesting, but the trouble with these small built in accelerators is that they were mostly designed when YOLO was the pinnacle of edge applications

by rcarmo 3y ago

This looks interesting, but the trouble with these small built in accelerators is that they were mostly designed when YOLO was the pinnacle of edge applications. These days they’re grossly underpowered…

tucosan 3y ago
Plus, they have way too little memory for any meaningful LLM workloads that go beyond simple inference tasks or running yolo.
bradfa 3y ago
Grossly underpowered compared to a fat stonking GPU, yes. But the whole BeagleY-AI board probably consumes less power at full tilt than just the fans to keep a modern 2-slot NVIDIA GPU cool. For embedded applications, the C7x DSPs are mighty. Lots of existing C6x DSP code can easily be recompiled to target the C7x with minimal effort as well, which is a big deal. Making your C6x code work efficiently on the C7x may require more than just a recompile, but being able to leverage existing C6x DSP codebases with minimal investment while getting the extra performance of the C7x is a very big deal. I don't think a quad-core ARM Cortex-A53 with a pair of C7x DSPs is trying to compete with an NVIDIA Hopper. But if you're making a $100 embedded product that has to handle video, this seems quite attractive.
- rcarmo 3y ago
  I’ve been trying to use rknn on RK3588 boards (which can go a bit higher in wattage due to the rest of the SOC and peripherals), and I get your point, but the overall landscape for non-GPU accelerators is still pretty much… just plain bad. I’m hoping for Ryzen APUs to be a good stopgap for larger models, or for enough support for Mali GPUs (which can add a little more compute) to be more usable, but in general, you have a huge abyss between “oh, that’s a face in that picture” and “here’s my current estimation of movement for this human”.
teaearlgraycold 3y ago
What do the cool kids use these days? I know YOLO is still relevant but I’m not sure what’s the new stuff.
- rcarmo 3y ago
  Well, I’ve been playing around with RK3588 boards. YOLO is still their reference example, and rknn and TFlite are just… too challenging to use for anything else, really. (I’ve been looking at timeseries and audio stuff, so I can sort of butcher my models to fit, but it’s still too small. And a GPU is still too power-hungry, etc. Am hoping for Ryzen APUs to be a useful stopgap.)

by rcarmo 3y ago

tucosan 3y ago

Plus, they have way too little memory for any meaningful LLM workloads that go beyond simple inference tasks or running yolo.

bradfa 3y ago

Grossly underpowered compared to a fat stonking GPU, yes. But the whole BeagleY-AI board probably consumes less power at full tilt than just the fans to keep a modern 2-slot NVIDIA GPU cool. For embedded applications, the C7x DSPs are mighty. Lots of existing C6x DSP code can easily be recompiled to target the C7x with minimal effort as well, which is a big deal. Making your C6x code work efficiently on the C7x may require more than just a recompile, but being able to leverage existing C6x DSP codebases with minimal investment while getting the extra performance of the C7x is a very big deal. I don't think a quad-core ARM Cortex-A53 with a pair of C7x DSPs is trying to compete with an NVIDIA Hopper. But if you're making a $100 embedded product that has to handle video, this seems quite attractive.

rcarmo 3y ago
I’ve been trying to use rknn on RK3588 boards (which can go a bit higher in wattage due to the rest of the SOC and peripherals), and I get your point, but the overall landscape for non-GPU accelerators is still pretty much… just plain bad. I’m hoping for Ryzen APUs to be a good stopgap for larger models, or for enough support for Mali GPUs (which can add a little more compute) to be more usable, but in general, you have a huge abyss between “oh, that’s a face in that picture” and “here’s my current estimation of movement for this human”.

teaearlgraycold 3y ago

What do the cool kids use these days? I know YOLO is still relevant but I’m not sure what’s the new stuff.

rcarmo 3y ago
Well, I’ve been playing around with RK3588 boards. YOLO is still their reference example, and rknn and TFlite are just… too challenging to use for anything else, really. (I’ve been looking at timeseries and audio stuff, so I can sort of butcher my models to fit, but it’s still too small. And a GPU is still too power-hungry, etc. Am hoping for Ryzen APUs to be a useful stopgap.)