2 ms·
GPUs are still software programmable. An "LLM chip" does not need that and so can be much more efficient.
by amelius 7mo ago
GPUs are still software programmable.
An "LLM chip" does not need that and so can be much more efficient.
- angoragoats 7mo agoSure, but that’s somewhat orthogonal to the point I was making, which is that LLMs are huge in size. Even in the case of a custom “LLM chip,” you’ll need huge amounts of very fast storage of some sort (likely DRAM), which places constraints on the size, power consumption, and cost of such a device. This device, if it existed, would not in any way resemble the Coral TPU product that the GP was referencing; I think in fact it would be closer in size, price, and form factor to a GPU.