4 ms·

Personally I think we're doing AI wrong. Every time we simplify the architecture we get huge improvement in speed. CPUs have to everything (load/store, branchin

Personally I think we're doing AI wrong. Every time we simplify the architecture we get huge improvement in speed. CPUs have to everything (load/store, branching, execution), GPUs do a lot less branching and aren't optimised for it. This is why they can be made massively parallel. What if we cut out arbitrary load/store too? This would look like A DSP. All memory would be prepared in advance and the massively parallel accelerator would "stream walk" through It without having to load store arbitrary memory locations. The speed improvement could be on par with CPU->Gpu. Of course this would be for inference mostly. There is a startup that is trying to do just that called tinycorp. I'm certainly watching it with interest

jorgemf 3y ago
As far as I know you can preload the data in the GPU before processing it. What is the difference/advantage of what you are proposing?
- Filligree 3y ago
  You wouldn't need the hardware to support arbitrary load/stores. In particular, you could get rid of (some of) the address lines... I'm unsure if this would be much of a win.
  - jorgemf 3y ago
    well, the data is way smaller than the model (at least the current trend), and you probably still need random access for the weights of the model. I am not sure if it a gain worth to pursue.
the8472 3y ago
> All memory would be prepared in advance and the massively parallel accelerator would "stream walk" through It without having to load store arbitrary memory locations. Dedicated accelerators do that already, e.g. google's TPUs, tesla's D1 or apple's neural engine. You must load the data into compute-unit local memory first before executing matmuls. Keeping the weights there and only piping the dynamic data through it saves memory bandwidth.
- pclmulqdq 3y ago
  This sort of architecture only really works when the network is small enough, which has been a perennial problem for neural network accelerators as networks grow but accelerators don't. LLMs and the like will often prefer the opposite form of "stream walking" (streaming the weights through the data) or a hybrid.
  - dhruvdh 3y ago
    Streams weights instead of data sounds really interesting - I had never considered it. Something else that might be theoretically possible is - Large array of FPGAs are apparently used to simulate and verify chips [1], can the same be done to run LLMs? Can we have 0.25 to 1 token per cycle, would the engineering effort be worth it, and would it be financially feasible from a TCO standpoint? [1] https://www.servethehome.com/amd-vp1902-is-leviathan-fpga-doubling-the-previous-gen-largest-fpga/ https://www.servethehome.com/amd-vp1902-is-leviathan-fpga-do...
    - pclmulqdq 3y ago
      The only problem is that an array of FPGAs would make an Nvidia GPU look cheap. And easy to get.
    - buildbot 3y ago
      People have used FPGAs for DNNs before https://www.microsoft.com/en-us/research/project/project-brainwave/ https://www.microsoft.com/en-us/research/project/project-bra...
    - vGPU 3y ago
      FGPA’s have been tried and dismissed. GPU’s are simply better with speed and efficiency.
sp332 3y ago
Itanium tried removing a bunch of the out-of-order hardware and hoped that compilers could schedule everything in advance. Generally, that did not work very well.
- subharmonicon 3y ago
  I don’t get the sense that this is what the parent comment is talking about at all. Not to mention that GPUs already execute in-order (at least any that I’m familiar with). They do have multiple execution pipelines, but instruction fetch/decode is in-order unlike something like a typical modern high performance CPU.
jacquesm 3y ago
There have been some research projects in this direction (computing fabrics, in-memory computation) but nothing that I'm aware of that made it to larger scale manufacturing.
shwaj 3y ago
This sounds a lot like the thesis behind George Hotz’s tinygrad, and the accelerators he eventually hopes to design for it. edit: typo

jorgemf 3y ago

As far as I know you can preload the data in the GPU before processing it. What is the difference/advantage of what you are proposing?

Filligree 3y ago
You wouldn't need the hardware to support arbitrary load/stores. In particular, you could get rid of (some of) the address lines... I'm unsure if this would be much of a win.
- jorgemf 3y ago
  well, the data is way smaller than the model (at least the current trend), and you probably still need random access for the weights of the model. I am not sure if it a gain worth to pursue.

the8472 3y ago

> All memory would be prepared in advance and the massively parallel accelerator would "stream walk" through It without having to load store arbitrary memory locations. Dedicated accelerators do that already, e.g. google's TPUs, tesla's D1 or apple's neural engine. You must load the data into compute-unit local memory first before executing matmuls. Keeping the weights there and only piping the dynamic data through it saves memory bandwidth.

pclmulqdq 3y ago
This sort of architecture only really works when the network is small enough, which has been a perennial problem for neural network accelerators as networks grow but accelerators don't. LLMs and the like will often prefer the opposite form of "stream walking" (streaming the weights through the data) or a hybrid.
- dhruvdh 3y ago
  Streams weights instead of data sounds really interesting - I had never considered it. Something else that might be theoretically possible is - Large array of FPGAs are apparently used to simulate and verify chips [1], can the same be done to run LLMs? Can we have 0.25 to 1 token per cycle, would the engineering effort be worth it, and would it be financially feasible from a TCO standpoint? [1] https://www.servethehome.com/amd-vp1902-is-leviathan-fpga-doubling-the-previous-gen-largest-fpga/ https://www.servethehome.com/amd-vp1902-is-leviathan-fpga-do...
  - pclmulqdq 3y ago
    The only problem is that an array of FPGAs would make an Nvidia GPU look cheap. And easy to get.
  - buildbot 3y ago
    People have used FPGAs for DNNs before https://www.microsoft.com/en-us/research/project/project-brainwave/ https://www.microsoft.com/en-us/research/project/project-bra...
  - vGPU 3y ago
    FGPA’s have been tried and dismissed. GPU’s are simply better with speed and efficiency.

sp332 3y ago

Itanium tried removing a bunch of the out-of-order hardware and hoped that compilers could schedule everything in advance. Generally, that did not work very well.

subharmonicon 3y ago
I don’t get the sense that this is what the parent comment is talking about at all. Not to mention that GPUs already execute in-order (at least any that I’m familiar with). They do have multiple execution pipelines, but instruction fetch/decode is in-order unlike something like a typical modern high performance CPU.

jacquesm 3y ago

There have been some research projects in this direction (computing fabrics, in-memory computation) but nothing that I'm aware of that made it to larger scale manufacturing.

shwaj 3y ago

This sounds a lot like the thesis behind George Hotz’s tinygrad, and the accelerators he eventually hopes to design for it. edit: typo