4 ms·

Could you really expect ASICs to increase performance exponentially if Moore's law is dead? As for new algorithms, surely this would be more of a logarithmic i

by swexbe 3y ago

Could you really expect ASICs to increase performance exponentially if Moore's law is dead? As for new algorithms, surely this would be more of a logarithmic increase for any specific problem as it gets harder and harder to improve on existing knowledge. Has, say, sorting become exponentially faster since Tony Hoare invented quicksort in 1959? My guess is no.

jiggawatts 3y ago
the NVIDIA A100 and H100 GPUs being used to run the GPT models are general purpose devices. If they had video ports, you could run 3D games on them! They're like a CPU: highly programmable, flexible, multi-purpose. This means that when any single "part" of the GPT inference code is running, it's probably using only 10-40% of the die area while the rest is not used and idle. Imagine a future where a nearly-optimal architecture for LLMs has been figured out and stops changing every few months. At that point, it would be possible to make a dedicated chip that does "nothing else but that". It would be 100% utilised and faster because it would have more units dedicated to the specific task at hand. The data paths could also be optimised to minimise wasteful trips to-and-from external memory, etc... The ideal "inference chip" would be something akin to an FPGA. Huge numbers of simple cells that can do just the dozen-or-so instructions required, with local memory for the weights and biases. Make these on a cheaper process, run them at 500 MHz for power efficiency, and tile them out by their hundreds in a hybrid memory+compute architecture. It would be relatively easy to get a factor of 10x, and I suspect 100x is possible.
buildbot 3y ago
Absolutely - the sibling comment gave a good rundown, but you could even go farther and to make an ASIC with a completely baked in model. Like an old text to speech chip kinda.

4 ms·

Could you really expect ASICs to increase performance exponentially if Moore's law is dead? As for new algorithms, surely this would be more of a logarithmic i

by swexbe 3y ago

jiggawatts 3y ago
the NVIDIA A100 and H100 GPUs being used to run the GPT models are general purpose devices. If they had video ports, you could run 3D games on them! They're like a CPU: highly programmable, flexible, multi-purpose. This means that when any single "part" of the GPT inference code is running, it's probably using only 10-40% of the die area while the rest is not used and idle. Imagine a future where a nearly-optimal architecture for LLMs has been figured out and stops changing every few months. At that point, it would be possible to make a dedicated chip that does "nothing else but that". It would be 100% utilised and faster because it would have more units dedicated to the specific task at hand. The data paths could also be optimised to minimise wasteful trips to-and-from external memory, etc... The ideal "inference chip" would be something akin to an FPGA. Huge numbers of simple cells that can do just the dozen-or-so instructions required, with local memory for the weights and biases. Make these on a cheaper process, run them at 500 MHz for power efficiency, and tile them out by their hundreds in a hybrid memory+compute architecture. It would be relatively easy to get a factor of 10x, and I suspect 100x is possible.
buildbot 3y ago
Absolutely - the sibling comment gave a good rundown, but you could even go farther and to make an ASIC with a completely baked in model. Like an old text to speech chip kinda.