6 ms·

I'm not a hardware guy, so this question is rather naive. How to they run programs on it? With a PC+GPU system you can just run the Python + tensorflow/pytorc

by zapnuk 7y ago

I'm not a hardware guy, so this question is rather naive. How to they run programs on it? With a PC+GPU system you can just run the Python + tensorflow/pytorch/etc. code. All required drivers can be easily installed. How to they do it with a custom co-processor?

_visgean 7y ago
To my understanding when you run tensorflow it builds the computational graph in representation that can be processed either on CPU or GPU, so for example it seems that here you would need "compiler" that takes the graph and runs it on the device.. similar to Edge TPU. So essentially you would install drivers and then communicate through standard interface.. But I might be wrong...
- pjscott 7y ago
  Yes, they have a bunch of special model compilation stuff that needs to happen in order for the ML model to run on their hardware, such as weight quantization, and then they'd also need to generate the a program for the NPU so it knows what to do -- what to convolve with what, and so on.
traverseda 7y ago
The co-processor has some kind of IO. On a GPU that IO is generally a PCIE port of some kind. Once you have the two machines able to talk to one another on the physical level, you need to decide what kind of protocol they're going to use. The simplest example of that whole flow is serial ports, which you may be familiar with if you've ever worked with arduino or microcontrollers or other embedded systems. So basically the answer is "the same way any two computers talk to one another", a protocol carried over some kind of physical medium. At the point you have to create your own drivers, and your own way of compiling code to run on the co-processor, but fundamentally it's no different from loading an arduino sketch onto a micro-controller. There are also solutions involving shared memory, where all the processors/co-processors are connected to the same ram chips. Those generally aren't practical with x86 chips, although it's a popular way for baseboard processors to talk to the ARM chips that run cellphones. It's also ultimately they way "symmetric multiprocessing" works in every computer that has more than one core on it's CPU. It's also probably how the "neural processor" CPU communicates with all the custom silicone that actually enables this product. It's likely they have a conventional microcontroller core that handles the input stream from the host computer and puts that data into the RAM layout required by the "NPU". What they've labeled the "front end". I haven't really looked into their architecture in any depth though.
- w8vY7ER 7y ago
  Thanks for what I thought was a really nice explanation for the uninitiated! Certainly helped me conceptualize it better.
- dragontamer 7y ago
  PCIe is probably the "standard" connection that I'd expect in a system today. 16x lanes of PCIe 3.0 will provide over 15 GBps of bandwidth at ~microsecond of latency. PCIe 3.0 as a protocol is complex enough to handle atomics these days (atomic-compare and swap), which allows for high-speed coordination and collaboration, very much akin to multi-core programming on cpu-to-cpu. "Exotic" connections go beyond PCIe. You could theoretically share DDR4 RAM. With this, its probably more natural to have "cache-coherent" interconnects. Assuming the MESI protocol (aka: very much simplified), the coprocessor can raise a section of RAM (64 bytes or so) into the "exclusive" state. As long as the coprocessor is holding exclusive, the CPU will stall as it tries to read that RAM. The co-processor can return the memory to the CPU control by setting it to "Invalid" state. The CPU can then read (or write) to memory by setting the memory to exclusive: under the CPU's control. Communication to and from the co-processor in this case isn't "memory-mapped IO", as much as it is "I/O over memory". :-). My explanation above is simplified, but hopefully you get the gist. A realistic example of a coherency-fabric is something like POWER9 OpenCAPI can communicate with NVidia GPUs at over 50GBps, while sharing the same memory space. (PCIe 3.0 and PCIe 4.0 has features to allow a similar effect at slightly slower speeds. I would bet that PCIe 5.0 is working on some kind of coherency fabric, but I don't actually know). The most exotic would probably be iGPUs (Intel) and APUs (AMD), which perform this kind of cache-coherent communication but at the L3 cache level and below. Since APUs and iGPUs share the same chip, they can send these messages back and forth without ever leaving the chip. Just entirely within L3 cache itself.
  - traverseda 7y ago
    Very interesting. I've seen a similar idea proposed to enable shared-memory based RPC, instead of communicating over a socket both processes agree to "trade" a chunk of memory back and forth, using it as a buffer for RPC messages.
pjc50 7y ago
Write some custom code. Probably in assembler, because it's an unusual special-purpose architecture. > The ISA consists of instructions with up to 4 slots with complex values. There are only eight instructions in total – two DMA read and write, three dot-product operations, and scaling and element-wise addition. That's not something you're going to target with a standard compiler. It doesn't even have any control flow instructions. Effectively they've built a "convolve machine": all they have to do is set up the addresses for the areas of memory containing the image and the weights, let it run, and collect the result.

by zapnuk 7y ago

_visgean 7y ago

To my understanding when you run tensorflow it builds the computational graph in representation that can be processed either on CPU or GPU, so for example it seems that here you would need "compiler" that takes the graph and runs it on the device.. similar to Edge TPU. So essentially you would install drivers and then communicate through standard interface.. But I might be wrong...

pjscott 7y ago
Yes, they have a bunch of special model compilation stuff that needs to happen in order for the ML model to run on their hardware, such as weight quantization, and then they'd also need to generate the a program for the NPU so it knows what to do -- what to convolve with what, and so on.

traverseda 7y ago

The co-processor has some kind of IO. On a GPU that IO is generally a PCIE port of some kind. Once you have the two machines able to talk to one another on the physical level, you need to decide what kind of protocol they're going to use. The simplest example of that whole flow is serial ports, which you may be familiar with if you've ever worked with arduino or microcontrollers or other embedded systems. So basically the answer is "the same way any two computers talk to one another", a protocol carried over some kind of physical medium. At the point you have to create your own drivers, and your own way of compiling code to run on the co-processor, but fundamentally it's no different from loading an arduino sketch onto a micro-controller. There are also solutions involving shared memory, where all the processors/co-processors are connected to the same ram chips. Those generally aren't practical with x86 chips, although it's a popular way for baseboard processors to talk to the ARM chips that run cellphones. It's also ultimately they way "symmetric multiprocessing" works in every computer that has more than one core on it's CPU. It's also probably how the "neural processor" CPU communicates with all the custom silicone that actually enables this product. It's likely they have a conventional microcontroller core that handles the input stream from the host computer and puts that data into the RAM layout required by the "NPU". What they've labeled the "front end". I haven't really looked into their architecture in any depth though.

w8vY7ER 7y ago
Thanks for what I thought was a really nice explanation for the uninitiated! Certainly helped me conceptualize it better.
dragontamer 7y ago
PCIe is probably the "standard" connection that I'd expect in a system today. 16x lanes of PCIe 3.0 will provide over 15 GBps of bandwidth at ~microsecond of latency. PCIe 3.0 as a protocol is complex enough to handle atomics these days (atomic-compare and swap), which allows for high-speed coordination and collaboration, very much akin to multi-core programming on cpu-to-cpu. "Exotic" connections go beyond PCIe. You could theoretically share DDR4 RAM. With this, its probably more natural to have "cache-coherent" interconnects. Assuming the MESI protocol (aka: very much simplified), the coprocessor can raise a section of RAM (64 bytes or so) into the "exclusive" state. As long as the coprocessor is holding exclusive, the CPU will stall as it tries to read that RAM. The co-processor can return the memory to the CPU control by setting it to "Invalid" state. The CPU can then read (or write) to memory by setting the memory to exclusive: under the CPU's control. Communication to and from the co-processor in this case isn't "memory-mapped IO", as much as it is "I/O over memory". :-). My explanation above is simplified, but hopefully you get the gist. A realistic example of a coherency-fabric is something like POWER9 OpenCAPI can communicate with NVidia GPUs at over 50GBps, while sharing the same memory space. (PCIe 3.0 and PCIe 4.0 has features to allow a similar effect at slightly slower speeds. I would bet that PCIe 5.0 is working on some kind of coherency fabric, but I don't actually know). The most exotic would probably be iGPUs (Intel) and APUs (AMD), which perform this kind of cache-coherent communication but at the L3 cache level and below. Since APUs and iGPUs share the same chip, they can send these messages back and forth without ever leaving the chip. Just entirely within L3 cache itself.
- traverseda 7y ago
  Very interesting. I've seen a similar idea proposed to enable shared-memory based RPC, instead of communicating over a socket both processes agree to "trade" a chunk of memory back and forth, using it as a buffer for RPC messages.

pjc50 7y ago

Write some custom code. Probably in assembler, because it's an unusual special-purpose architecture. > The ISA consists of instructions with up to 4 slots with complex values. There are only eight instructions in total – two DMA read and write, three dot-product operations, and scaling and element-wise addition. That's not something you're going to target with a standard compiler. It doesn't even have any control flow instructions. Effectively they've built a "convolve machine": all they have to do is set up the addresses for the areas of memory containing the image and the weights, let it run, and collect the result.