5 ms·

There are very different architectures in the wild. Some are simply standard GPUs (maybe with additional support for bf16/float16) (Rockchip RK1808 has one like

by phh 3y ago

There are very different architectures in the wild. Some are simply standard GPUs (maybe with additional support for bf16/float16) (Rockchip RK1808 has one like that). You give it a list of instructions pretty much like a CPU (except massively parallel), and it'll execute it. BTW when I say standard GPU, I'm not saying "kinda like GPU, but really literally GPU architecture. Linux mainline support for Amlogic A311D2's NPU is 10 lines, and it's declaring a no-output vivante GPU. Some are just hardware pipeline to compute 2d/3d convolutions + activation function (Rockchip RK3588 has one like that). You give it the memory address + dimensions of the input matrix, the memory address + dimensions of the weights, memory address + dimensions of the output, which activation function you want (there are only like 4 supported), then you tell it RUN, you wait for a bit, and you have the result at the output memory address. (I took Rockchip example to show that even in one microcosm it can change a lot) And then you can imagine any architecture in-between. AFAIK they all work with some RAM as input and some RAM as output, but some may have its own RAM, some may share RAM with the system, some might have mixed usage (RK3588 has some SRAM, and when you ask it to compute the convolution, you can tell it either to write to SRAM or system RAM) It's possible that there are some components that are border line between ISP (Image Signal Porcessing) and a NPU, where the input is the direct camera stream, but my guess is that they do some very small processing on the camera stream, then dump it to RAM, then do all the heavy work from RAM to RAM. I think that Pixel 4-5 had something like that.

grozzle 3y ago
The RK3588 seems like a beast for performance per dollar, I have one running desktop Linux for emulators and games, but I haven't used the NPU yet, because I haven't taken the time to figure out how to get OpenCV to talk to it. Is Rockchip's stuff "good", as NPUs go? I'm thinking of buying another 3588 SoC for my robotics hobby - you seem like you'd know if that's a decent idea or not.
- reesul 3y ago
  Yeah it’s performance vs cost is honestly nuts. 6 TOPS is a pretty solid NPU, but I don’t know what their software is like. Programming those accelerators is often difficult, especially if you’re a small time customer. Curious if anyone can weigh in on their SW usability. A quick search for their user level tools showed examples/documentation in Chinese(?)
  - grozzle 3y ago
    Yeah it's the first under $100 system I've tried, (and I'm a fan of the genre) that's truly a desktop replacement in terms of being a snappy responsive desktop even with lots of browser tabs, etc. I do know they have their own special sauce to talk to the NPU. I was discouraged from making the effort myself because their special sauce to talk to the VPU has barely any ffmpeg support, it p much only uses gstreamer, and I'm neither a masochist nor French so that's a non-starter.
    - reesul 3y ago
      Yeah I’m very surprised to see a single board computer at less than $100 for a processor like that. Hard to tell what their actual 1ku price is, but if the random Alibaba I found for $20 is right, then that price for the overall board is absurd. By VPU are you talking about stuff like ISP, video encoder/decoder, or something else? Among embedded processors I’ve seen touting vision acceleration, gstreamer support is fairly widespread. I bit the bullet to learn it because my role requires it. Maybe it’s Stockholm Syndrome talking, but I’ve somehow grown to like gstreamer. The learning curve was awkward. I struggled with documentation and learned more by analyzing some examples and trial-and-error.
      - grozzle 3y ago
        By VPU, I meant the square on the board that eats h.265 hw_enc and hw_dec and things like that, yes. I've been told it's a separate square from the GPU, by someone who is full-time waist-deep on getting Arch running on that hardware, so I take it as fact. Oh no. The prospect of gstreamer being the only way... Oh no. Maybe there's a zsh plugin or smth that autocompletes sane defaults? AAAAAAAAHHHHHH there surely isn't, anyone merciful enough to make one would just use ffmpeg instead...
- phh 3y ago
  Well, I already said it :P RK3588's NPU can only do convolutions and a small list of activation functions. Realistically it's usable only on image (2D, 3D data). The software side is pretty weird: It's amazing the amount of models they support despite the limited fixed-hardware function, but there has been literally 0 development in the last 6 months and it does have a lot of bugs that doesn't seem hard to fix. Even when it comes to images, it doesn't support changing the resolution of the input (you need to ""recompile the model"" for that), which is super weird since hardware pipeline doesn't care much about the size. Anyway, I really don't recommend it, unless you're making your own model, you know before-hand what's supported and what isn't, and your input is fixed resolution (which is a pretty fair usage in an embedded system) (fixed = doesn't change at every frame. handling hotplug from one webcam with a resolution to another with another resolution is fine) I think looking at the examples give you a reasonable show of what it can do: https://github.com/rockchip-linux/rknn-toolkit/tree/master/examples https://github.com/rockchip-linux/rknn-toolkit/tree/master/e... It's mobilenet, yolov3, resnet50. There aren't more examples because they didn't had more examples. There aren't more examples because that's pretty much all you can reasonably run. As far as I can tell, modern image models using transformer/vit won't be runnable on it. (it acts enough as a coprocessor that it's possible to do some parts in CPU some parts in NPU - and Rockchip framework handles that -, so maybe it's somehow possible) (Note: I say this as a huge Rockchip lover, their mainline support is top-notch, they make very durable product (their 2015's RK3288 is still far from obsolete), and I bought a RK3588 SBC to play with a NPU accelerator (whose full specification is publicly available btw), in the hope to have a self-hosted LLM voice assistant)

by phh 3y ago

grozzle 3y ago

The RK3588 seems like a beast for performance per dollar, I have one running desktop Linux for emulators and games, but I haven't used the NPU yet, because I haven't taken the time to figure out how to get OpenCV to talk to it. Is Rockchip's stuff "good", as NPUs go? I'm thinking of buying another 3588 SoC for my robotics hobby - you seem like you'd know if that's a decent idea or not.

reesul 3y ago
Yeah it’s performance vs cost is honestly nuts. 6 TOPS is a pretty solid NPU, but I don’t know what their software is like. Programming those accelerators is often difficult, especially if you’re a small time customer. Curious if anyone can weigh in on their SW usability. A quick search for their user level tools showed examples/documentation in Chinese(?)
- grozzle 3y ago
  Yeah it's the first under $100 system I've tried, (and I'm a fan of the genre) that's truly a desktop replacement in terms of being a snappy responsive desktop even with lots of browser tabs, etc. I do know they have their own special sauce to talk to the NPU. I was discouraged from making the effort myself because their special sauce to talk to the VPU has barely any ffmpeg support, it p much only uses gstreamer, and I'm neither a masochist nor French so that's a non-starter.
  - reesul 3y ago
    Yeah I’m very surprised to see a single board computer at less than $100 for a processor like that. Hard to tell what their actual 1ku price is, but if the random Alibaba I found for $20 is right, then that price for the overall board is absurd. By VPU are you talking about stuff like ISP, video encoder/decoder, or something else? Among embedded processors I’ve seen touting vision acceleration, gstreamer support is fairly widespread. I bit the bullet to learn it because my role requires it. Maybe it’s Stockholm Syndrome talking, but I’ve somehow grown to like gstreamer. The learning curve was awkward. I struggled with documentation and learned more by analyzing some examples and trial-and-error.
    - grozzle 3y ago
      By VPU, I meant the square on the board that eats h.265 hw_enc and hw_dec and things like that, yes. I've been told it's a separate square from the GPU, by someone who is full-time waist-deep on getting Arch running on that hardware, so I take it as fact. Oh no. The prospect of gstreamer being the only way... Oh no. Maybe there's a zsh plugin or smth that autocompletes sane defaults? AAAAAAAAHHHHHH there surely isn't, anyone merciful enough to make one would just use ffmpeg instead...
phh 3y ago
Well, I already said it :P RK3588's NPU can only do convolutions and a small list of activation functions. Realistically it's usable only on image (2D, 3D data). The software side is pretty weird: It's amazing the amount of models they support despite the limited fixed-hardware function, but there has been literally 0 development in the last 6 months and it does have a lot of bugs that doesn't seem hard to fix. Even when it comes to images, it doesn't support changing the resolution of the input (you need to ""recompile the model"" for that), which is super weird since hardware pipeline doesn't care much about the size. Anyway, I really don't recommend it, unless you're making your own model, you know before-hand what's supported and what isn't, and your input is fixed resolution (which is a pretty fair usage in an embedded system) (fixed = doesn't change at every frame. handling hotplug from one webcam with a resolution to another with another resolution is fine) I think looking at the examples give you a reasonable show of what it can do: https://github.com/rockchip-linux/rknn-toolkit/tree/master/examples https://github.com/rockchip-linux/rknn-toolkit/tree/master/e... It's mobilenet, yolov3, resnet50. There aren't more examples because they didn't had more examples. There aren't more examples because that's pretty much all you can reasonably run. As far as I can tell, modern image models using transformer/vit won't be runnable on it. (it acts enough as a coprocessor that it's possible to do some parts in CPU some parts in NPU - and Rockchip framework handles that -, so maybe it's somehow possible) (Note: I say this as a huge Rockchip lover, their mainline support is top-notch, they make very durable product (their 2015's RK3288 is still far from obsolete), and I bought a RK3588 SBC to play with a NPU accelerator (whose full specification is publicly available btw), in the hope to have a self-hosted LLM voice assistant)