4 ms·

A PCIe Coral TPU Finally Works on Raspberry Pi 5

114 points by mikece 3y ago

jauntywundrkind 3y ago
Coral is 4 years old, and it's both shocking & not that there aren't many competitors out there today. Also a bit sad PyCoral requires python 3.9! Yikes!
- filterfiber 3y ago
  Especially since it only has like 8 MB of memory I think?
  - geerlingguy 3y ago
    Some SoCs even have competent built in NPUs, too, but software support is severely lacking. PyCoral and Coral hardware development seems glacial lately :( They have enjoyed a lot of momentum... I wish they could release a follow-up version and get some longstanding software issues resolved.
    - deleted 3y ago
      [deleted]
    - orangepurple 3y ago
      Orange Pi 5 has RK3588 with a neural processing unit (NPU). SDKs for it https://github.com/rockchip-linux/rknpu2 https://github.com/rockchip-linux/rknpu2 https://github.com/rockchip-linux/rknn-toolkit2 https://github.com/rockchip-linux/rknn-toolkit2 Quickstart and usage guide: https://wiki.t-firefly.com/en/ROC-RK3588S-PC/usage_npu.html https://wiki.t-firefly.com/en/ROC-RK3588S-PC/usage_npu.html
      - ThatPlayer 3y ago
        One of the most popular uses I've seen for Coral, Frigate for object detection for security cameras, recently added support for the RK3588 NPUs too: https://github.com/blakeblackshear/frigate/pull/8382 https://github.com/blakeblackshear/frigate/pull/8382 . I think it requiring a RK3588 makes it a bit annoying to have to upgrade your entire setup just for this though.
        orangepurple 3y ago
        How many camera feeds can one Orange Pi 5 realistically monitor?
        ThatPlayer 3y ago
        According to the author of that PR, they're using 10% of 1 NPU core on 3 cameras: https://github.com/blakeblackshear/frigate/pull/8382#issuecomment-1785953476 https://github.com/blakeblackshear/frigate/pull/8382#issueco... The bottleneck instead will probably be the video stream decoding speed, especially as the SoC's hardware decoder isn't being used yet.
- gymbeaux 3y ago
  I think companies prefer to develop AI stuff and then sell it, rather than sell the hardware.
  - nmstoker 3y ago
    However keeping the Python version up to date shouldn't be that hard though, should it?
    - gymbeaux 3y ago
      3.9 isn’t that bad on my opinion. I work with other Python libraries, popular ones, that won’t work past 3.9. I think PyPDF is one of them? Something with PDF parsing.
- muxamilian 3y ago
  The Hailo (https://hailo.ai https://hailo.ai) is the biggest competitor at the moment. It is a lot more powerful while not requiring much more power. It does, however, have some problems with Pyhon's GIL and documentation is not always perfect. Other than that it's like the Coral but much better.
  - franciscop 3y ago
    Any alternative that has a simple USB connector?
  - deleted 3y ago
    [deleted]
  - geerlingguy 3y ago
    It's also a bit more expensive :) But I am going to test one of those out soon, too! Hopefully it's software stack is maintained a bit more actively than Coral
    - LoganDark 3y ago
      By "a bit more expensive", you mean $25,000 as opposed to Coral's $25? Because they will not even tell me the price without me filling out a big form.
    - muxamilian 3y ago
      The software is actively maintained yes. The price is between 100 to 200$.
- bragr 3y ago
  >Also a bit sad PyCoral requires python 3.9! Yikes! It's a bit on the old side, but hardly yikes territory. Ubuntu 20.04 ships with 3.8 [1] as the system python with support until April 2025, and AWS supports 3.7-3.11 for lambda runtimes [2]. 3.12 has only just been released [1] https://packages.ubuntu.com/focal/python3 https://packages.ubuntu.com/focal/python3 [2] https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes...
  - mianos 3y ago
    It needs a whole bunch binary bindings. It is not really supported since Ubuntu 18 or pretty much any other operating system. If you look at the github repo and bug tracker you will see it's largely abandoned. (I have a Coral USB and you can get it running with an old operating system version).
    - bragr 3y ago
      Well that's an entirely separate issue than the python version
      - mianos 3y ago
        Yes, that is my point. It's not really a python version issue.
westurner 3y ago
An HBM3E HAT would or would not yet make TPUs more useful with a Raspberry Pi 5? Jetson Nano (~$149) Orin Nano (~$499, 32 tensor cores, 40 TOPS) AGX Orin (200-275 TOPS) NVIDIA Jetson > Origins: https://en.wikipedia.org/wiki/Nvidia_Jetson#Versions https://en.wikipedia.org/wiki/Nvidia_Jetson#Versions TOPS for NVIDIA [Orin] Nano [AGX] https://connecttech.com/jetson/jetson-module-comparison/ https://connecttech.com/jetson/jetson-module-comparison/ Coral Mini-PCIe ($25; ? tensor cores, 4 TOPS (int8); 2 TOPS per watt) TPUv5 (393 TOPS) Tensor Processing Unit (TPU) https://en.wikipedia.org/wiki/Tensor_Processing_Unit https://en.wikipedia.org/wiki/Tensor_Processing_Unit AI Accelerator > Nomenclature: https://en.wikipedia.org/wiki/AI_accelerator https://en.wikipedia.org/wiki/AI_accelerator NVIDIA DLSS > Architecture: https://en.wikipedia.org/wiki/Deep_learning_super_sampling#Architecture https://en.wikipedia.org/wiki/Deep_learning_super_sampling#A... : > DLSS is only available on GeForce RTX 20, GeForce RTX 30, GeForce RTX 40, and Quadro RTX series of video cards, using dedicated AI accelerators called Tensor Cores. [23][28] Tensor Cores are available since the Nvidia Volta GPU microarchitecture, which was first used on the Tesla V100 line of products.[29] They are used for doing fused multiply-add (FMA) operations that are used extensively in neural network calculations for applying a large series of multiplications on weights, followed by the addition of a bias. Tensor cores can operate on FP16, INT8, INT4, and INT1 data types. Vision processing unit: https://en.wikipedia.org/wiki/Vision_processing_unit https://en.wikipedia.org/wiki/Vision_processing_unit Versatile Processor Unit (VPU)
byyoung3 3y ago
ive got some pcie corals if u want to buy one. lmk
- LoganDark 3y ago
  What are your asking prices?
  - byyoung3 3y ago
    make me an offer.
    - LoganDark 3y ago
      I'm not sure exactly what pcie corals you're referring to. If you mean the m.2 ones, it'd have to be less than the $25 or so MSRP, but if you have full-size cards I'd be very curious to see what they even look like first.
      - gymbeaux 3y ago
        My guess is they’re referring to the mini PCIe cards. I think Coral only comes in NVMe and mini PCIe (and technically you can buy just the chip itself I believe). Obviously mini PCIe is deprecated and it’s unlikely many people have a device with a mini PCIe slot AND they don’t need that slot for WiFi/BT. That’s probably why they’re looking to sell.
        LoganDark 3y ago
        The adapter for this would cost more than the device itself, so I don't think I can use it. I'd take an NVMe module in a heartbeat though.
qwertox 3y ago
Makes me wonder how Radxa Rock 5B's NPU can be used: - Supporting INT4 / INT8 / INT16 / FP16 / BF16 and TF32 acceleration - Computing power is up to 6TOPs

114 points by mikece 3y ago

Coral is 4 years old, and it's both shocking & not that there aren't many competitors out there today. Also a bit sad PyCoral requires python 3.9! Yikes!

filterfiber 3y ago
Especially since it only has like 8 MB of memory I think?
- geerlingguy 3y ago
  Some SoCs even have competent built in NPUs, too, but software support is severely lacking. PyCoral and Coral hardware development seems glacial lately :( They have enjoyed a lot of momentum... I wish they could release a follow-up version and get some longstanding software issues resolved.
  - deleted 3y ago
    [deleted]
  - orangepurple 3y ago
    Orange Pi 5 has RK3588 with a neural processing unit (NPU). SDKs for it https://github.com/rockchip-linux/rknpu2 https://github.com/rockchip-linux/rknpu2 https://github.com/rockchip-linux/rknn-toolkit2 https://github.com/rockchip-linux/rknn-toolkit2 Quickstart and usage guide: https://wiki.t-firefly.com/en/ROC-RK3588S-PC/usage_npu.html https://wiki.t-firefly.com/en/ROC-RK3588S-PC/usage_npu.html
    - ThatPlayer 3y ago
      One of the most popular uses I've seen for Coral, Frigate for object detection for security cameras, recently added support for the RK3588 NPUs too: https://github.com/blakeblackshear/frigate/pull/8382 https://github.com/blakeblackshear/frigate/pull/8382 . I think it requiring a RK3588 makes it a bit annoying to have to upgrade your entire setup just for this though.
      - orangepurple 3y ago
        How many camera feeds can one Orange Pi 5 realistically monitor?
        ThatPlayer 3y ago
        According to the author of that PR, they're using 10% of 1 NPU core on 3 cameras: https://github.com/blakeblackshear/frigate/pull/8382#issuecomment-1785953476 https://github.com/blakeblackshear/frigate/pull/8382#issueco... The bottleneck instead will probably be the video stream decoding speed, especially as the SoC's hardware decoder isn't being used yet.
gymbeaux 3y ago
I think companies prefer to develop AI stuff and then sell it, rather than sell the hardware.
- nmstoker 3y ago
  However keeping the Python version up to date shouldn't be that hard though, should it?
  - gymbeaux 3y ago
    3.9 isn’t that bad on my opinion. I work with other Python libraries, popular ones, that won’t work past 3.9. I think PyPDF is one of them? Something with PDF parsing.
muxamilian 3y ago
The Hailo (https://hailo.ai https://hailo.ai) is the biggest competitor at the moment. It is a lot more powerful while not requiring much more power. It does, however, have some problems with Pyhon's GIL and documentation is not always perfect. Other than that it's like the Coral but much better.
- franciscop 3y ago
  Any alternative that has a simple USB connector?
- deleted 3y ago
  [deleted]
- geerlingguy 3y ago
  It's also a bit more expensive :) But I am going to test one of those out soon, too! Hopefully it's software stack is maintained a bit more actively than Coral
  - LoganDark 3y ago
    By "a bit more expensive", you mean $25,000 as opposed to Coral's $25? Because they will not even tell me the price without me filling out a big form.
  - muxamilian 3y ago
    The software is actively maintained yes. The price is between 100 to 200$.
bragr 3y ago
>Also a bit sad PyCoral requires python 3.9! Yikes! It's a bit on the old side, but hardly yikes territory. Ubuntu 20.04 ships with 3.8 [1] as the system python with support until April 2025, and AWS supports 3.7-3.11 for lambda runtimes [2]. 3.12 has only just been released [1] https://packages.ubuntu.com/focal/python3 https://packages.ubuntu.com/focal/python3 [2] https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes...
- mianos 3y ago
  It needs a whole bunch binary bindings. It is not really supported since Ubuntu 18 or pretty much any other operating system. If you look at the github repo and bug tracker you will see it's largely abandoned. (I have a Coral USB and you can get it running with an old operating system version).
  - bragr 3y ago
    Well that's an entirely separate issue than the python version
    - mianos 3y ago
      Yes, that is my point. It's not really a python version issue.

westurner 3y ago

An HBM3E HAT would or would not yet make TPUs more useful with a Raspberry Pi 5? Jetson Nano (~$149) Orin Nano (~$499, 32 tensor cores, 40 TOPS) AGX Orin (200-275 TOPS) NVIDIA Jetson > Origins: https://en.wikipedia.org/wiki/Nvidia_Jetson#Versions https://en.wikipedia.org/wiki/Nvidia_Jetson#Versions TOPS for NVIDIA [Orin] Nano [AGX] https://connecttech.com/jetson/jetson-module-comparison/ https://connecttech.com/jetson/jetson-module-comparison/ Coral Mini-PCIe ($25; ? tensor cores, 4 TOPS (int8); 2 TOPS per watt) TPUv5 (393 TOPS) Tensor Processing Unit (TPU) https://en.wikipedia.org/wiki/Tensor_Processing_Unit https://en.wikipedia.org/wiki/Tensor_Processing_Unit AI Accelerator > Nomenclature: https://en.wikipedia.org/wiki/AI_accelerator https://en.wikipedia.org/wiki/AI_accelerator NVIDIA DLSS > Architecture: https://en.wikipedia.org/wiki/Deep_learning_super_sampling#Architecture https://en.wikipedia.org/wiki/Deep_learning_super_sampling#A... : > DLSS is only available on GeForce RTX 20, GeForce RTX 30, GeForce RTX 40, and Quadro RTX series of video cards, using dedicated AI accelerators called Tensor Cores. [23][28] Tensor Cores are available since the Nvidia Volta GPU microarchitecture, which was first used on the Tesla V100 line of products.[29] They are used for doing fused multiply-add (FMA) operations that are used extensively in neural network calculations for applying a large series of multiplications on weights, followed by the addition of a bias. Tensor cores can operate on FP16, INT8, INT4, and INT1 data types. Vision processing unit: https://en.wikipedia.org/wiki/Vision_processing_unit https://en.wikipedia.org/wiki/Vision_processing_unit Versatile Processor Unit (VPU)

byyoung3 3y ago

ive got some pcie corals if u want to buy one. lmk

LoganDark 3y ago
What are your asking prices?
- byyoung3 3y ago
  make me an offer.
  - LoganDark 3y ago
    I'm not sure exactly what pcie corals you're referring to. If you mean the m.2 ones, it'd have to be less than the $25 or so MSRP, but if you have full-size cards I'd be very curious to see what they even look like first.
    - gymbeaux 3y ago
      My guess is they’re referring to the mini PCIe cards. I think Coral only comes in NVMe and mini PCIe (and technically you can buy just the chip itself I believe). Obviously mini PCIe is deprecated and it’s unlikely many people have a device with a mini PCIe slot AND they don’t need that slot for WiFi/BT. That’s probably why they’re looking to sell.
      - LoganDark 3y ago
        The adapter for this would cost more than the device itself, so I don't think I can use it. I'd take an NVMe module in a heartbeat though.

qwertox 3y ago

Makes me wonder how Radxa Rock 5B's NPU can be used: - Supporting INT4 / INT8 / INT16 / FP16 / BF16 and TF32 acceleration - Computing power is up to 6TOPs