5 ms·

Nvidia Deep Learning Accelerator (NVDLA): free open inference accelerator (2017)

62 points by my123 6y ago

blintz 6y ago
The cynic in me is confused. Nvidia isn’t exactly known for making lots of things open source. What’s their angle here? Is the idea to take steam out of projects like TPU by making viable, open-source alternatives?
- sliken 6y ago
  Seems pretty straight forward to me. Their maximum profit comes from selling more GPUs, instead of trying to sell a piece of software that has open source competition.
  - tpmx 6y ago
    It's not like their sales people are having a hard time pushing the hardware at the moment. (In case someone hasn't noticed: All GPUs have been sold out for months.)
    - sliken 6y ago
      Sure, but they have to think ahead. GPUs make sense for ethereum, but not bitcoin (which moved from GPUs to FPGAs to ASICs). Ethereum is moving from proof of work to proof of stake. I expect the used market to flood with used mining GPUs once that happens.
- modeless 6y ago
  Commoditize your complement. Nvidia wants to sell neural net training chips. Neural net inference chips are a complement.
- tasty_freeze 6y ago
  > Nvidia isn’t exactly known for making lots of things open source. It is true nvidia drivers are not open source, but this github page says nvidia has 248 repositories: https://github.com/NVIDIA https://github.com/NVIDIA I'm not sure why the link was submitted today. The repo says it is 3-4 years old.
  - kkielhofner 6y ago
    I understand why Nvidia gets a lot of heat from the Linux community regarding the closed source nature of their GPU drivers. That said, I use them on all of my Linux machines and not being a FOSS purist I don't take issue with it. Additionally, most of the complaints come from the Linux desktop community. You rarely see the ML community at large expressing any concern. For the most part they're just trying to get work done and the overall CUDA ecosystem is tremendously useful in that regard. Very usable desktop experience, CUDA just works, the rest of the software ecosystem makes anything else pale by comparison (see the Pytorch ROCm discussion that made the HN front page today). Every few months I take another look at ROCm and frankly don't understand how anyone can take it seriously. In the FOSS/Linux community the discussion around the closed source nature of desktop drivers seriously discounts the amount of FOSS work Nvidia has done.
    - pirocks 6y ago
      My perception is that Nvidia in particular gets a lot of heat because they have bad/buggy drivers which integrate poorly with existing package managers/distros, not just closed source drivers. Anecdotally I have at least one issue on my home desktop which is suspect is caused by Nvidia drivers and poor display hotswap on a work device which is definetly caused by Nvidia.
      - kkielhofner 6y ago
        That's fair but every time the issue is raised on HN, for example, there are anecdotal reports all over the place. Given the ridiculous number of variables I will believe virtually anything to be true for that specific person, configuration, use case, etc. I understand the support for AMD's in kernel drivers (and Mesa), rooting for the underdog, etc but anyone who is being honest will acknowledge there are plenty of issues with their drivers and various configurations as well. Assuming it's six of one, half dozen of the other the fact that Nvidia drivers also bring the entire CUDA ecosystem to any GPU made in the last five years (or more) really tips the scales (overall) for the justification of an Nvidia hardware purchase (all other things being equal).
- fortran77 6y ago
  NVIDIA has a lot of open source. https://github.com/NVIDIA https://github.com/NVIDIA NVIDIA's business is to sell GPUs. This helps.
shmerl 6y ago
> a free and open architecture that promotes a standard way to design deep learning inference accelerators. Is it really open and standard, i.e. not using anything like CUDA?
justicezyx 6y ago
A componentized DL inference hardware library + compiler suite + inference runtime. Overall already appears a solid foundation for a production system ATM. Another example of nVidia's unchallenged prowess in DL software ecosystem. The hardware design library could be seen as a prominent example of "software defined hardware". Specifically, the compnentized hardware library dissects DL inference into 4 high-level operations: convolution, pooling, activation, normalization. Hardware description code is provided for these 4 operations. Allowing practitioners to tailor their chip design to their application. For instance, one could remove an operation from the chip, and adjust the size and throughput of any one operation. Software: compiler suite, and runtime are provided for building and running actual DL models. Compiler suite also operate based on the 4 operations, and produce an IR format that models around the operations; and then optimized towards the target chip. Runtime include kernel module and user space driver to handle the setup of the chip runtime and accept user request of loading model and inference requests. 2 questions I did not find explicit answer: * Is there an inference-aware training capability? * Does the compiler suite also consider the actual hardware spec during its optimization?
sradman 6y ago
NVDLA [1] targets IoT devices: > NVDLA is available for product development as part of NVIDIA's Jetson Xavier NX, a small circuit board in a form factor about the size of a credit card which includes a 6-core ARMv8.2 64-bit CPU, an integrated 384-core Volta GPU with 48 Tensor Cores, and dual NVDLA "engines", as described in their own press release. NVIDIA claims the product will deliver 14 TOPS (tera operations per second) of compute under 10 W, but most of this likely comes from the GPU cores. Applications broadly include edge computing inference engines, including object recognition for autonomous driving. > NVIDIA's involvement with open hardware includes the use of RISC-V processors as part of their GPU product line-up. [1] https://en.wikipedia.org/wiki/NVDLA https://en.wikipedia.org/wiki/NVDLA
- my123 6y ago
  The BeagleV (https://beagleboard.org/beaglev https://beagleboard.org/beaglev) also has an NVDLA in the SoC. (1 core w/ 2048 MACs at 800MHz)
mikehollinger 6y ago
I'm actually kind of confused here; NVDLA has been around for a while. What is new or interesting here that changes anything? I did a cursory look to see if there's any new announcements, but - even things like the roadmap on the site show a 2017 roadmap, and the last commits to the hw and sw repos are 2019. Perhaps this needs an edit to say it's from... 2017? Or is it 2019?
- rjsw 6y ago
  I only learned about NVDLA recently, from reading the specs of the BeagleV board that another poster has linked to.
- bostonpete 6y ago
  Should definitely say (2017). Looking at the Wayback Machine from November 2017, I don't notice any differences from the current page.
  - dang 6y ago
    Added. Thanks!
    - mikehollinger 6y ago
      Cool beans. :-)
halotrope 6y ago
Is there any information on how to source components outside Nvidias Jetson ecosystem? It would be interesting to be able to buy such chip individually for e.g. embedded projects but there seem no vendors listed. Considering the DLA project is not particularly new I wonder if it will remain a niche appliance on Nvidias boards.
lights0123 6y ago
In the license: > “Other NVDLA Rights” includes copyright, design right (whether registered or unregistered), semiconductor topography (mask work) rights, and database rights to the NVDLA Specification and any Derivative Work. For the avoidance of doubt, Other NVDLA Rights does not include patents or trademarks. > Subject to the terms and conditions of this License, NVIDIA and each Contributor hereby grant to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable license under the Other NVDLA Rights to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the NVDLA Specification and such Derivative Works, and to commercially exploit any mask works included in the NVDLA Specification or such Derivative Works Why would they grant you permissions to "semiconductor topography" rights?
- zokier 6y ago
  Because of this I imagine? https://en.wikipedia.org/wiki/Semiconductor_Chip_Protection_Act_of_1984 https://en.wikipedia.org/wiki/Semiconductor_Chip_Protection_...
  - lights0123 6y ago
    Right, but why would Nvidia want to give you the rights to their silicon designs?
ojn 6y ago
This is 2+ years old, and the Github has seen zero updates. Seems like a dead-end effort to me.
- 6pac3rings 6y ago
  The Microsoft buying Github isn't an encouragement to contribute for some people knowing what they know of previous Microsoft leadership.

62 points by my123 6y ago

The cynic in me is confused. Nvidia isn’t exactly known for making lots of things open source. What’s their angle here? Is the idea to take steam out of projects like TPU by making viable, open-source alternatives?

sliken 6y ago
Seems pretty straight forward to me. Their maximum profit comes from selling more GPUs, instead of trying to sell a piece of software that has open source competition.
- tpmx 6y ago
  It's not like their sales people are having a hard time pushing the hardware at the moment. (In case someone hasn't noticed: All GPUs have been sold out for months.)
  - sliken 6y ago
    Sure, but they have to think ahead. GPUs make sense for ethereum, but not bitcoin (which moved from GPUs to FPGAs to ASICs). Ethereum is moving from proof of work to proof of stake. I expect the used market to flood with used mining GPUs once that happens.
modeless 6y ago
Commoditize your complement. Nvidia wants to sell neural net training chips. Neural net inference chips are a complement.
tasty_freeze 6y ago
> Nvidia isn’t exactly known for making lots of things open source. It is true nvidia drivers are not open source, but this github page says nvidia has 248 repositories: https://github.com/NVIDIA https://github.com/NVIDIA I'm not sure why the link was submitted today. The repo says it is 3-4 years old.
- kkielhofner 6y ago
  I understand why Nvidia gets a lot of heat from the Linux community regarding the closed source nature of their GPU drivers. That said, I use them on all of my Linux machines and not being a FOSS purist I don't take issue with it. Additionally, most of the complaints come from the Linux desktop community. You rarely see the ML community at large expressing any concern. For the most part they're just trying to get work done and the overall CUDA ecosystem is tremendously useful in that regard. Very usable desktop experience, CUDA just works, the rest of the software ecosystem makes anything else pale by comparison (see the Pytorch ROCm discussion that made the HN front page today). Every few months I take another look at ROCm and frankly don't understand how anyone can take it seriously. In the FOSS/Linux community the discussion around the closed source nature of desktop drivers seriously discounts the amount of FOSS work Nvidia has done.
  - pirocks 6y ago
    My perception is that Nvidia in particular gets a lot of heat because they have bad/buggy drivers which integrate poorly with existing package managers/distros, not just closed source drivers. Anecdotally I have at least one issue on my home desktop which is suspect is caused by Nvidia drivers and poor display hotswap on a work device which is definetly caused by Nvidia.
    - kkielhofner 6y ago
      That's fair but every time the issue is raised on HN, for example, there are anecdotal reports all over the place. Given the ridiculous number of variables I will believe virtually anything to be true for that specific person, configuration, use case, etc. I understand the support for AMD's in kernel drivers (and Mesa), rooting for the underdog, etc but anyone who is being honest will acknowledge there are plenty of issues with their drivers and various configurations as well. Assuming it's six of one, half dozen of the other the fact that Nvidia drivers also bring the entire CUDA ecosystem to any GPU made in the last five years (or more) really tips the scales (overall) for the justification of an Nvidia hardware purchase (all other things being equal).
fortran77 6y ago
NVIDIA has a lot of open source. https://github.com/NVIDIA https://github.com/NVIDIA NVIDIA's business is to sell GPUs. This helps.

shmerl 6y ago

> a free and open architecture that promotes a standard way to design deep learning inference accelerators. Is it really open and standard, i.e. not using anything like CUDA?

justicezyx 6y ago

A componentized DL inference hardware library + compiler suite + inference runtime. Overall already appears a solid foundation for a production system ATM. Another example of nVidia's unchallenged prowess in DL software ecosystem. The hardware design library could be seen as a prominent example of "software defined hardware". Specifically, the compnentized hardware library dissects DL inference into 4 high-level operations: convolution, pooling, activation, normalization. Hardware description code is provided for these 4 operations. Allowing practitioners to tailor their chip design to their application. For instance, one could remove an operation from the chip, and adjust the size and throughput of any one operation. Software: compiler suite, and runtime are provided for building and running actual DL models. Compiler suite also operate based on the 4 operations, and produce an IR format that models around the operations; and then optimized towards the target chip. Runtime include kernel module and user space driver to handle the setup of the chip runtime and accept user request of loading model and inference requests. 2 questions I did not find explicit answer: * Is there an inference-aware training capability? * Does the compiler suite also consider the actual hardware spec during its optimization?

sradman 6y ago

NVDLA [1] targets IoT devices: > NVDLA is available for product development as part of NVIDIA's Jetson Xavier NX, a small circuit board in a form factor about the size of a credit card which includes a 6-core ARMv8.2 64-bit CPU, an integrated 384-core Volta GPU with 48 Tensor Cores, and dual NVDLA "engines", as described in their own press release. NVIDIA claims the product will deliver 14 TOPS (tera operations per second) of compute under 10 W, but most of this likely comes from the GPU cores. Applications broadly include edge computing inference engines, including object recognition for autonomous driving. > NVIDIA's involvement with open hardware includes the use of RISC-V processors as part of their GPU product line-up. [1] https://en.wikipedia.org/wiki/NVDLA https://en.wikipedia.org/wiki/NVDLA

my123 6y ago
The BeagleV (https://beagleboard.org/beaglev https://beagleboard.org/beaglev) also has an NVDLA in the SoC. (1 core w/ 2048 MACs at 800MHz)

mikehollinger 6y ago

I'm actually kind of confused here; NVDLA has been around for a while. What is new or interesting here that changes anything? I did a cursory look to see if there's any new announcements, but - even things like the roadmap on the site show a 2017 roadmap, and the last commits to the hw and sw repos are 2019. Perhaps this needs an edit to say it's from... 2017? Or is it 2019?

rjsw 6y ago
I only learned about NVDLA recently, from reading the specs of the BeagleV board that another poster has linked to.
bostonpete 6y ago
Should definitely say (2017). Looking at the Wayback Machine from November 2017, I don't notice any differences from the current page.
- dang 6y ago
  Added. Thanks!
  - mikehollinger 6y ago
    Cool beans. :-)

halotrope 6y ago

Is there any information on how to source components outside Nvidias Jetson ecosystem? It would be interesting to be able to buy such chip individually for e.g. embedded projects but there seem no vendors listed. Considering the DLA project is not particularly new I wonder if it will remain a niche appliance on Nvidias boards.

lights0123 6y ago

In the license: > “Other NVDLA Rights” includes copyright, design right (whether registered or unregistered), semiconductor topography (mask work) rights, and database rights to the NVDLA Specification and any Derivative Work. For the avoidance of doubt, Other NVDLA Rights does not include patents or trademarks. > Subject to the terms and conditions of this License, NVIDIA and each Contributor hereby grant to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable license under the Other NVDLA Rights to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the NVDLA Specification and such Derivative Works, and to commercially exploit any mask works included in the NVDLA Specification or such Derivative Works Why would they grant you permissions to "semiconductor topography" rights?

zokier 6y ago
Because of this I imagine? https://en.wikipedia.org/wiki/Semiconductor_Chip_Protection_Act_of_1984 https://en.wikipedia.org/wiki/Semiconductor_Chip_Protection_...
- lights0123 6y ago
  Right, but why would Nvidia want to give you the rights to their silicon designs?

ojn 6y ago

This is 2+ years old, and the Github has seen zero updates. Seems like a dead-end effort to me.

6pac3rings 6y ago
The Microsoft buying Github isn't an encouragement to contribute for some people knowing what they know of previous Microsoft leadership.