5 ms·

Excuse my ignorance, but what sort of applications benefit from hardware acceleration on limited SoC resources?

by ttymck 3y ago

sosodev 3y ago
Inferencing neural networks is a big one. It's hard to get near real-time performance without it. Depending on the use case of course.
- chaxor 3y ago
  Why on earth would you run ML on a potato?
  - bkirkby 3y ago
    inference, object detection for one.
  - microtherion 3y ago
    To give one example, Raspberry Pi are often used to control 3D printers and attached webcams, and there are ML systems that supervise the webcam feed for evidence of a print failure. Being able to run these systems on the SBC would be an advantage, but that's typically not realistic today.
  - schreiaj 3y ago
    What should we run it on if we want to deploy it on mobile systems with limited access to network and extremely restricted power budgets? For a hobby project I'm trying to find a solution - Power budget for multiple is sub 200w. Need to run inference on a lower resolution video stream (or multiple, that be nice) to do object detection. Cost is a factor because I need to have multiple angles to determine where in relation to a mobile platform the target object is. I'm looking at the Coral.ai board because RPi like boards lack the ability to do ML tasks at reasonable FPS and NVidia seems to have abandoned the lower cost side of the market since the Jetson Nanos seem to be less and less available. (Not that Coral.ai boards are available at all...)
    - Eisenstein 3y ago
      Check out the Luxonis Oak products. I use an Oak-1 Lite to do real time 2 stage object detection and recognition (~23FPS at 1080P inference on device with two custom yolov5n models). With a bit of python and a Pi (or a Rock64 or similar) you can get it up and running in a day. They also have a decent community and are actively developing the API/SDK and hardware.
      - schreiaj 3y ago
        Thanks, I've got one of their depth cameras that's been ok. I didn't realize they'd expanded their line so much. Glad to hear about the API/SDK improving, last time I mucked with it a year or so ago it seemed like it was underdeveloped. Going to have to dig into the sensors they use - had passable luck with non ML tasks using dirt cheap camera modules from laptops running at low resolutions right up until I started moving the cameras at all and then it became a blurry mess because they were so small their exposure times were high. (I'm trying to also avoid having to put a bunch of illumination near the cameras so it doesn't entirely look like a biblically accurate angel)
        Eisenstein 3y ago
        Well, you can control the camera ISP and they use very decent IMX modules, so it really shouldn't an issue like it was with the cheapos, as long as you can do the coding to your needs. EDIT: * https://docs.luxonis.com/projects/api/en/latest/samples/ColorCamera/rgb_camera_control/ https://docs.luxonis.com/projects/api/en/latest/samples/Colo...
        eisen 3y ago
        Check out TI’s new low cost board vision AI board: https://www.ti.com/tool/SK-AM62A-LP https://www.ti.com/tool/SK-AM62A-LP They have a home grown AI accelerator along with free Deep learning SDK. Also offer a pretty easy online tool (free again) to use called TI Edge AI studio. They are using extending the existing AI solutions that come from the higher performance parts like TDA4 and AM68A parts. Pretty good considering a lot of these manufactures are just buying other AI Ip that isn’t performing great and investing in their own engineering.
  - oneplane 3y ago
    It would probably not run on this potato directly, but on an accelerator (be it in-SoC or a separate thing like a Google Coral). The management and control of a Coral needs to happen on a CPU somewhere, but the actual work doesn't involve the CPU much, if at all. So if you have a thing that has all the data and all the work done on an engine elsewhere and you just need to have a SoC to turn the thing on and off and get the data in and out, that's where a potato-SoC could work. Of course, the potato would need good distro support with up-to-date kernel, drivers, python, libraries etc. and if you're connecting it to the network, best make sure it's also getting patched consistently. So, as far as ML Potatoes go, that's about it. It is a totally valid question by the way, even if asked in jest.
  - happymellon 3y ago
    Why wouldn't you? It's the same situation as all the bitcoin miners from years ago. They mined on the GPU, so the CPU didn't matter as much as how many PCIe slots the motherboard had. The CPU is purely there to drive the operating system and enable the network to communicate with the GPU. If you aren't using the CPU for anything other than the PCIe slot to enable a Nvidia card for ML, then a very cheap potato makes a lot of sense.
oneplane 3y ago
You're actually answering your own question here ;-) Because a SoC with limited CPU-core resources can't do everything in software, the chip contains many system components (hence System-on-Chip or System-on-a-Chip) that handle things the CPU cores then no longer need to do. Think protocol handling or memory; instead of spending many clock cycles on handling the USB bus, you can leave that to the USB controller and only deal with what is actually relevant to your USB device. Same with the VOP (Video Output Processor) block, instead of spending many clock cycles on putting the right bits in the frame buffer you tell the VOP that you'd like the background to be orange and then only spend time setting the right bits for black text (for example). So instead of having to deal with many millions of bits, you only have to deal with less than 1% of them because every bit you don't set becomes orange. For other things like I2C, I2S, DMA, networking, cryptography, SD-IO, GPIO, PWM etc. the same applies. Instead of constantly spending time setting the right bit at the right time many times each second, you just tell a dedicated block on the SoC to do a thing in a certain pattern and it will do it for you, consuming on CPU core resources. This also allows slow CPU cores that wouldn't be able to decode video in real time to offload the entire decoding to a video decoder block, and then tell the GPU part of the SoC that you're drawing a green rectangle somewhere and that's where it has to put the decoded video frames. Why would one do all of this? Because it's cheap and power-efficient, and that's how you make a big pile of money. I have no idea how accurate or up-to-date this PDF is, but it should at least give you some idea as to what a SoC can do without bogging down the CPU cores: https://dl.radxa.com/rockpis/docs/hw/datasheets/Rockchip%20RK3308TRM%20V1.1%20Part2-20180810.pdf https://dl.radxa.com/rockpis/docs/hw/datasheets/Rockchip%20R... Check chapter 9 for example, all of those boxes are things you don't have to spend CPU cycles on. If you did use the CPU, it would be super slow.
- jancsika 3y ago
  Is this in the purview of the panfrost driver stack?
  - oneplane 3y ago
    I think that's only for ARM's own GPU IP, not for the VOP (it's more of a LCD driver than an actual GPU -- I'd compare the VOP to say, GameBoy-color level of graphics capabilities, which is cool but not even close to a bare-bones VESA or UEFI framebuffer).

Excuse my ignorance, but what sort of applications benefit from hardware acceleration on limited SoC resources?

by ttymck 3y ago

Excuse my ignorance, but what sort of applications benefit from hardware acceleration on limited SoC resources?

Inferencing neural networks is a big one. It's hard to get near real-time performance without it. Depending on the use case of course.

chaxor 3y ago
Why on earth would you run ML on a potato?
- bkirkby 3y ago
  inference, object detection for one.
- microtherion 3y ago
  To give one example, Raspberry Pi are often used to control 3D printers and attached webcams, and there are ML systems that supervise the webcam feed for evidence of a print failure. Being able to run these systems on the SBC would be an advantage, but that's typically not realistic today.
- schreiaj 3y ago
  What should we run it on if we want to deploy it on mobile systems with limited access to network and extremely restricted power budgets? For a hobby project I'm trying to find a solution - Power budget for multiple is sub 200w. Need to run inference on a lower resolution video stream (or multiple, that be nice) to do object detection. Cost is a factor because I need to have multiple angles to determine where in relation to a mobile platform the target object is. I'm looking at the Coral.ai board because RPi like boards lack the ability to do ML tasks at reasonable FPS and NVidia seems to have abandoned the lower cost side of the market since the Jetson Nanos seem to be less and less available. (Not that Coral.ai boards are available at all...)
  - Eisenstein 3y ago
    Check out the Luxonis Oak products. I use an Oak-1 Lite to do real time 2 stage object detection and recognition (~23FPS at 1080P inference on device with two custom yolov5n models). With a bit of python and a Pi (or a Rock64 or similar) you can get it up and running in a day. They also have a decent community and are actively developing the API/SDK and hardware.
    - schreiaj 3y ago
      Thanks, I've got one of their depth cameras that's been ok. I didn't realize they'd expanded their line so much. Glad to hear about the API/SDK improving, last time I mucked with it a year or so ago it seemed like it was underdeveloped. Going to have to dig into the sensors they use - had passable luck with non ML tasks using dirt cheap camera modules from laptops running at low resolutions right up until I started moving the cameras at all and then it became a blurry mess because they were so small their exposure times were high. (I'm trying to also avoid having to put a bunch of illumination near the cameras so it doesn't entirely look like a biblically accurate angel)
      - Eisenstein 3y ago
        Well, you can control the camera ISP and they use very decent IMX modules, so it really shouldn't an issue like it was with the cheapos, as long as you can do the coding to your needs. EDIT: * https://docs.luxonis.com/projects/api/en/latest/samples/ColorCamera/rgb_camera_control/ https://docs.luxonis.com/projects/api/en/latest/samples/Colo...
        eisen 3y ago
        Check out TI’s new low cost board vision AI board: https://www.ti.com/tool/SK-AM62A-LP https://www.ti.com/tool/SK-AM62A-LP They have a home grown AI accelerator along with free Deep learning SDK. Also offer a pretty easy online tool (free again) to use called TI Edge AI studio. They are using extending the existing AI solutions that come from the higher performance parts like TDA4 and AM68A parts. Pretty good considering a lot of these manufactures are just buying other AI Ip that isn’t performing great and investing in their own engineering.
- oneplane 3y ago
  It would probably not run on this potato directly, but on an accelerator (be it in-SoC or a separate thing like a Google Coral). The management and control of a Coral needs to happen on a CPU somewhere, but the actual work doesn't involve the CPU much, if at all. So if you have a thing that has all the data and all the work done on an engine elsewhere and you just need to have a SoC to turn the thing on and off and get the data in and out, that's where a potato-SoC could work. Of course, the potato would need good distro support with up-to-date kernel, drivers, python, libraries etc. and if you're connecting it to the network, best make sure it's also getting patched consistently. So, as far as ML Potatoes go, that's about it. It is a totally valid question by the way, even if asked in jest.
- happymellon 3y ago
  Why wouldn't you? It's the same situation as all the bitcoin miners from years ago. They mined on the GPU, so the CPU didn't matter as much as how many PCIe slots the motherboard had. The CPU is purely there to drive the operating system and enable the network to communicate with the GPU. If you aren't using the CPU for anything other than the PCIe slot to enable a Nvidia card for ML, then a very cheap potato makes a lot of sense.

oneplane 3y ago

You're actually answering your own question here ;-) Because a SoC with limited CPU-core resources can't do everything in software, the chip contains many system components (hence System-on-Chip or System-on-a-Chip) that handle things the CPU cores then no longer need to do. Think protocol handling or memory; instead of spending many clock cycles on handling the USB bus, you can leave that to the USB controller and only deal with what is actually relevant to your USB device. Same with the VOP (Video Output Processor) block, instead of spending many clock cycles on putting the right bits in the frame buffer you tell the VOP that you'd like the background to be orange and then only spend time setting the right bits for black text (for example). So instead of having to deal with many millions of bits, you only have to deal with less than 1% of them because every bit you don't set becomes orange. For other things like I2C, I2S, DMA, networking, cryptography, SD-IO, GPIO, PWM etc. the same applies. Instead of constantly spending time setting the right bit at the right time many times each second, you just tell a dedicated block on the SoC to do a thing in a certain pattern and it will do it for you, consuming on CPU core resources. This also allows slow CPU cores that wouldn't be able to decode video in real time to offload the entire decoding to a video decoder block, and then tell the GPU part of the SoC that you're drawing a green rectangle somewhere and that's where it has to put the decoded video frames. Why would one do all of this? Because it's cheap and power-efficient, and that's how you make a big pile of money. I have no idea how accurate or up-to-date this PDF is, but it should at least give you some idea as to what a SoC can do without bogging down the CPU cores: https://dl.radxa.com/rockpis/docs/hw/datasheets/Rockchip%20RK3308TRM%20V1.1%20Part2-20180810.pdf https://dl.radxa.com/rockpis/docs/hw/datasheets/Rockchip%20R... Check chapter 9 for example, all of those boxes are things you don't have to spend CPU cycles on. If you did use the CPU, it would be super slow.

jancsika 3y ago
Is this in the purview of the panfrost driver stack?
- oneplane 3y ago
  I think that's only for ARM's own GPU IP, not for the VOP (it's more of a LCD driver than an actual GPU -- I'd compare the VOP to say, GameBoy-color level of graphics capabilities, which is cool but not even close to a bare-bones VESA or UEFI framebuffer).