4 ms·

The middle ground here would be an FPGA, but I belive you would need a very expensive one to implement an LLM on it.

• dogma1138 8mo ago
  FPGAs would be less efficient than GPUs. FPGAs don’t scale if they did all GPUs would’ve been replaced by FPGAs for graphics a long time ago. You use an FPGA when spinning a custom ASIC doesn’t makes financial sense and generic processor such as a CPU or GPU is overkill. Arguably the middle ground here are TPUs, just taking the most efficient parts of a “GPU” when it comes to these workloads but still relying on memory access in every step of the computation.
  • jgalt212 8mo ago
    I thought it was because the number logic elements in a GPU is orders of magnitude higher than in a FPGA, rather than just processing speed. And GPU processing is inherently parallel so the GPU beats the FPGA just based on transistor count.
    • dogma1138 8mo ago
      With FPGA you are sacrificing performance for flexibility you are far less efficient in transistors for any given task than with a dedicated ASIC even if it’s a general compute ASIC like a GPU is today. The reason no one is building large FPGAs is that there is no market for them. If an H200 scale FPGA was viable we would have one.