3 ms·

Is there any particular reason you'd want to use an FPGA for this? Unless your problem space is highly dynamic (e.g. prototyping) or you're making products in v

Is there any particular reason you'd want to use an FPGA for this? Unless your problem space is highly dynamic (e.g. prototyping) or you're making products in vanishing low quantities for a price insensitive market (e.g. military) an ASIC is always going to be better. There doesn't seem to be much flux in the low level architectures used for inferencing at this point, so may as well commit to an ASIC, as is already happening with Apple, Qualcomm, etc building NPUs into their SOCs.

someguydave 2y ago
gotta prototype the thing somewhere. If it turns out that the LLM algos become pretty mature I suspect accelerators of all kinds will be baked into silicon, especially for inference.
- jsheard 2y ago
  That's the thing though, we're already there. Every new consumer ARM and x86 ASIC is shipping with some kind of NPU, the time for tentatively testing the waters with FPGAs was a few years ago before this stuff came to market.
  - PaulHoule 2y ago
    But the NPU might be poorly designed for your model or workload or just poorly designed.
- mistrial9 2y ago
  like this? https://www.d-matrix.ai/product/ https://www.d-matrix.ai/product/
PaulHoule 2y ago
(1) Academics could make an FPGA but not an ASIC, (2) FPGA is a first step to make an ASIC
wongarsu 2y ago
This specific project looks like a case of "we have this platform for automotive and industrial use, running Llama on the dual-core ARM CPU is slow but there's an FPGA right next to it". That's all the justification you really need for a university project. Not sure how useful this is for anyone who isn't already locked into this specific architecture. But it might be a useful benchmark or jumping-off-point for more useful FPGA-based accelerators, like ones optimized for 1 bit or 1.58 bit LLMs
israrkhan 2y ago
You can open-source your FPGA designs for wider collaboration with the community? wider collaboration. Also, FPGA is the starting step to make any modern digital chip.
danielmarkbruce 2y ago
Model architecture changes fast. Maybe it will slow down.

someguydave 2y ago

gotta prototype the thing somewhere. If it turns out that the LLM algos become pretty mature I suspect accelerators of all kinds will be baked into silicon, especially for inference.

jsheard 2y ago
That's the thing though, we're already there. Every new consumer ARM and x86 ASIC is shipping with some kind of NPU, the time for tentatively testing the waters with FPGAs was a few years ago before this stuff came to market.
- PaulHoule 2y ago
  But the NPU might be poorly designed for your model or workload or just poorly designed.
mistrial9 2y ago
like this? https://www.d-matrix.ai/product/ https://www.d-matrix.ai/product/

PaulHoule 2y ago

(1) Academics could make an FPGA but not an ASIC, (2) FPGA is a first step to make an ASIC

wongarsu 2y ago

This specific project looks like a case of "we have this platform for automotive and industrial use, running Llama on the dual-core ARM CPU is slow but there's an FPGA right next to it". That's all the justification you really need for a university project. Not sure how useful this is for anyone who isn't already locked into this specific architecture. But it might be a useful benchmark or jumping-off-point for more useful FPGA-based accelerators, like ones optimized for 1 bit or 1.58 bit LLMs

israrkhan 2y ago

You can open-source your FPGA designs for wider collaboration with the community? wider collaboration. Also, FPGA is the starting step to make any modern digital chip.

danielmarkbruce 2y ago

Model architecture changes fast. Maybe it will slow down.