3 ms·

But the problem is not that your model is fast. Sure, you can go ASIC and go even faster, but the thing around GPU is that they scale well for both training an

by stevefan1999 2mo ago

But the problem is not that your model is fast. Sure, you can go ASIC and go even faster, but the thing around GPU is that they scale well for both training and inference, and the technical floor is low. The level to get into FPGA design is insanely high, you've got to read timing diagrams, you need to know combinatorial and sequential logics and good sense of boolean algebra, you need to have an asynchronous signal based mindset which is vastly different from CPU/GPU, you need to know netlist and you need to endure the time it takes for the EDA to finish generating it. Yosys is still years behind Xilinx There is a reason GPUs are called accelerators; it sacrifices and does not try to really specialize on one particular thing, except high parallel dataflow and branch-free calculation. Otherwise we will all be using DSPs

cgyvbunji 2mo ago
FPGAs are not power efficient at all vs GPUs and ASICs anyway, which is going to be especially true when they are fully saturated by LLM inference.
- stevefan1999 2mo ago
  That said, FPGA do provide a middle ground, but using it for speed and power efficient is not a forte, and the true value exactly comes from this focus alone: it allows you do emulate systhesis and verify that your logic is correct before you do full ASIC tapeout, e.g. building softcores for CPU validation Anything else is added and unintentional benefits.
  - deleted 2mo ago
    [deleted]
- adrian_b 2mo ago
  Nothing can be as power efficient as an ASIC, which is designed for a specific purpose, instead of being a programmable device intended to be suitable for a large class of applications. A GPU is much more efficient than an FPGA for what a GPU does. On the other hand for applications for which the set of primitive operations implemented in hardware by a GPU is not a good fit, an FPGA can be much more power efficient than a GPU. For applications that involve a massive amount of computations with FP32, FP16 or BF16 numbers, for which GPUs have special hardware execution units, i.e. for training and for inference with non-quantized models, there is no chance for an FPGA to be more efficient. If the GPU is recent enough to have good support for more heavily quantized data types, e.g. INT8, FP8, NVFP4 etc. an FPGA also does not have chances to be competitive. An FPGA could be more efficient than a GPU if either it is some special AI-oriented FPGA, which instead of having traditional arithmetic units oriented for DSP applications, has execution units implementing the quantized data types popular in ML/AI, or if it implements inference using some new not yet standardized data type, for which GPUs do not have dedicated support yet.
  - cgyvbunji 2mo ago
    Yeah but then it's basically an AI ASIC with an FPGA block inside it. Basically the less FPGA-like an FPGA is, ie the more dedicated silicon in the FPGA for the task in question, the more power efficient it is, because custom logic in an FPGA is done in LUTs which is RAM and RAM is way way more power hungry than actual logic gates, and the fabric is apparently power hungry too. It's unfortunate to me because I like FPGAs and wish they weren't so niche, but they are inherently limited in this way.
    - adrian_b 2mo ago
      I agree.
    - imtringued 2mo ago
      If you understood FPGAs you wouldn't think of them as incredibly niche. The more application specific you get, the smaller the total volume of chips. The very nature of application specifity ruins the economics of ASICs. Every time someone tells me an ASIC is more energy efficient I'm thinking, you just ruined the business case. The vast majority of application specific designs are not economically viable unless you use FPGAs to implement them.
      - deleted 2mo ago
        [deleted]
      - deleted 2mo ago
        [deleted]
      - cgyvbunji 2mo ago
        > The more application specific you get, the smaller the total volume of chips Sure, and that is the definition of niche. FPGAs are niche. You could have made a good point that an application specific design that already requires an FPGA might want to now also have a local LLM, so putting the LLM right on the FPGA might be the most expedient option in that case. Other than that, I don't think people are reaching for FPGAs to do LLM training or inference in general because I don't think it can be cost effective vs other options.
- imtringued 2mo ago
  I still don't know why anyone is obsessed with the theoretical performance over the practical/economical performance of the technologies in practice. FPGAs win against MCUs in terms of performance and they only lose in terms of static power consumption, not on performance per watt. Also, there was a company doing LLM inference on FPGAs and their entire selling point was that they were more energy efficient than Nvidia so you can add more FPGAs onto the same rack. Finally, the ASIC Vs FPGA battle is kind of meaningless because the moment you decide to reprogram your FPGA for any reason, ASICs aren't even in the same market anymore. Then there is the fact that FPGAs tends to have insane amounts of SRAM bandwidth compared to most chips.
IshKebab 2mo ago
FPGAs aren't that difficult to program. Waves / timing diagrams are trivial to understand, they can just be very tedious to read. The different execution model to CPUs also isn't very hard to understand IMO. Hardware people love to say that it is, but it really isn't. The hardest bit is probably SystemVerilog - it's just such a terrible language for hardware design. Full of footguns and gotchas and weird limitations and undocumented or tool-dependent stuff that you have to just know (like what is synthesizable). Approximately nobody uses Yosys.
- avmich 2mo ago
  I would recommend https://github.com/yuri-panchul/basics-graphics-music https://github.com/yuri-panchul/basics-graphics-music to those looking to learn FPGA.
- anitil 2mo ago
  Have you tried using HardCaml? I heard about it on the Jane Street (of course) podcast. I've only used vhdl and verilog, and liked neither
- imtringued 2mo ago
  Honestly, FPGAs are easy, but HLS is a deadend. The entire industry doubled down on HLS because of perverse incentives that encourage wasting FPGA resources. The goal of an FPGA vendor isn't to make FPGAs mainstream, no. Their goal is to sell you the biggest FPGA they can get away with even though from the customer's perspective you want to use the smallest FPGA possible to make the economics work.

cgyvbunji 2mo ago

FPGAs are not power efficient at all vs GPUs and ASICs anyway, which is going to be especially true when they are fully saturated by LLM inference.

stevefan1999 2mo ago
That said, FPGA do provide a middle ground, but using it for speed and power efficient is not a forte, and the true value exactly comes from this focus alone: it allows you do emulate systhesis and verify that your logic is correct before you do full ASIC tapeout, e.g. building softcores for CPU validation Anything else is added and unintentional benefits.
- deleted 2mo ago
  [deleted]
adrian_b 2mo ago
Nothing can be as power efficient as an ASIC, which is designed for a specific purpose, instead of being a programmable device intended to be suitable for a large class of applications. A GPU is much more efficient than an FPGA for what a GPU does. On the other hand for applications for which the set of primitive operations implemented in hardware by a GPU is not a good fit, an FPGA can be much more power efficient than a GPU. For applications that involve a massive amount of computations with FP32, FP16 or BF16 numbers, for which GPUs have special hardware execution units, i.e. for training and for inference with non-quantized models, there is no chance for an FPGA to be more efficient. If the GPU is recent enough to have good support for more heavily quantized data types, e.g. INT8, FP8, NVFP4 etc. an FPGA also does not have chances to be competitive. An FPGA could be more efficient than a GPU if either it is some special AI-oriented FPGA, which instead of having traditional arithmetic units oriented for DSP applications, has execution units implementing the quantized data types popular in ML/AI, or if it implements inference using some new not yet standardized data type, for which GPUs do not have dedicated support yet.
- cgyvbunji 2mo ago
  Yeah but then it's basically an AI ASIC with an FPGA block inside it. Basically the less FPGA-like an FPGA is, ie the more dedicated silicon in the FPGA for the task in question, the more power efficient it is, because custom logic in an FPGA is done in LUTs which is RAM and RAM is way way more power hungry than actual logic gates, and the fabric is apparently power hungry too. It's unfortunate to me because I like FPGAs and wish they weren't so niche, but they are inherently limited in this way.
  - adrian_b 2mo ago
    I agree.
  - imtringued 2mo ago
    If you understood FPGAs you wouldn't think of them as incredibly niche. The more application specific you get, the smaller the total volume of chips. The very nature of application specifity ruins the economics of ASICs. Every time someone tells me an ASIC is more energy efficient I'm thinking, you just ruined the business case. The vast majority of application specific designs are not economically viable unless you use FPGAs to implement them.
    - deleted 2mo ago
      [deleted]
    - deleted 2mo ago
      [deleted]
    - cgyvbunji 2mo ago
      > The more application specific you get, the smaller the total volume of chips Sure, and that is the definition of niche. FPGAs are niche. You could have made a good point that an application specific design that already requires an FPGA might want to now also have a local LLM, so putting the LLM right on the FPGA might be the most expedient option in that case. Other than that, I don't think people are reaching for FPGAs to do LLM training or inference in general because I don't think it can be cost effective vs other options.
imtringued 2mo ago
I still don't know why anyone is obsessed with the theoretical performance over the practical/economical performance of the technologies in practice. FPGAs win against MCUs in terms of performance and they only lose in terms of static power consumption, not on performance per watt. Also, there was a company doing LLM inference on FPGAs and their entire selling point was that they were more energy efficient than Nvidia so you can add more FPGAs onto the same rack. Finally, the ASIC Vs FPGA battle is kind of meaningless because the moment you decide to reprogram your FPGA for any reason, ASICs aren't even in the same market anymore. Then there is the fact that FPGAs tends to have insane amounts of SRAM bandwidth compared to most chips.

IshKebab 2mo ago

FPGAs aren't that difficult to program. Waves / timing diagrams are trivial to understand, they can just be very tedious to read. The different execution model to CPUs also isn't very hard to understand IMO. Hardware people love to say that it is, but it really isn't. The hardest bit is probably SystemVerilog - it's just such a terrible language for hardware design. Full of footguns and gotchas and weird limitations and undocumented or tool-dependent stuff that you have to just know (like what is synthesizable). Approximately nobody uses Yosys.

avmich 2mo ago
I would recommend https://github.com/yuri-panchul/basics-graphics-music https://github.com/yuri-panchul/basics-graphics-music to those looking to learn FPGA.
anitil 2mo ago
Have you tried using HardCaml? I heard about it on the Jane Street (of course) podcast. I've only used vhdl and verilog, and liked neither
imtringued 2mo ago
Honestly, FPGAs are easy, but HLS is a deadend. The entire industry doubled down on HLS because of perverse incentives that encourage wasting FPGA resources. The goal of an FPGA vendor isn't to make FPGAs mainstream, no. Their goal is to sell you the biggest FPGA they can get away with even though from the customer's perspective you want to use the smallest FPGA possible to make the economics work.