5 ms·
FPGAs will never rival gpus or TPUs for inference. The main reason is that GPUs aren't really gpus anymore. 50% of the die area or more is for fixed function ma
by fooblaster 9mo ago
FPGAs will never rival gpus or TPUs for inference. The main reason is that GPUs aren't really gpus anymore. 50% of the die area or more is for fixed function matrix multiplication units and associated dedicated storage. This just isn't general purpose anymore. FPGAs cannot rival this with their configurable DSP slices. They would need dedicated systolic blocks, which they aren't getting. The closest thing is the versal ML tiles, and those are entire peoxessors, not FPGA blocks. Those have failed by being impossible to program.
- ithkuil 9mo agoTurns out that a lot of interesting computation can be expressed as a matrix multiplication.
- fooblaster 9mo agoYeah, I wouldn't have guessed it would be helping me write systemverilog.
- alanma 9mo agoyup, GBs are so much tensor core nowadays :)
- Lerc 9mo agoI think it'll get to a point with quantisation that GPUs that run them will be more FPGA like than graphics renderers. If you quantize far enough things begin to look more like gates than floating point units. At that level a FPGA wouldn't run your model, it would be one your model.
- fpgaminer 9mo ago> FPGAs will never rival gpus or TPUs for inference. The main reason is that GPUs aren't really gpus anymore. Yeah. Even for Bitcoin mining GPUs dominated FPGAs. I created the Bitcoin mining FPGA project(s), and they were only interesting for two reasons: 1) they were far more power efficient, which in the case of mining changes the equation significantly. 2) GPUs at the time had poor binary math support, which hampered their performance; whereas an FPGA is just one giant binary math machine.
- beeflet 9mo agoI have wondered if it is possible to make a mining algorithm FPGA-hard in the same way that RandomX is CPU-hard and memory-hard. Relative to CPUs, the "programming time" cost is high. Nice username btw.
- hayley-patton 9mo agoMy recollection is that ASIC-resistance involves using lots of scratchpad memory and mixing multiple hashing algorithms, so that you'd have to use a lot of silicon and/or bottleneck hard on external RAM. I think the same would hurt FPGAs too.
- ksk23 9mo agoImho, not knowing too much bout both concepts; it kinda is! You would need to re-implement a general purpose cpu to beat it, or that was the idea behind RandomX
- dnautics 9mo agoI don't think this is correct. For inference, the bottleneck is memory bandwidth, so if you can hook up an FPGA with better memory, it has an outside shot at beating GPUs, at least in the short term. I mean, I have worked with FPGAs that outperform H200s in Llama3-class models a while and a half ago.
- fooblaster 9mo agoShow me a single FPGA that can outperform a B200 at matrix multiplication (or even come close) at any usable precision. B200 can do 10 peta ops at fp8, theoretically. I do agree memory bandwidth is also a problem for most FPGA setups, but xilinx ships HBM with some skus and they are not competitive at inference as far as I know.
- checker659 9mo agoSaid GPUs spend half the time just waiting for memory.
- fooblaster 9mo agoYep, but they are still 50x faster than any fpga.
- dnautics 9mo agoprobably not B200 level but better than you might expect: https://www.positron.ai/ https://www.positron.ai/ i believe a B200 is ~3x the H200 at llama-3, so that puts the FPGAs at around 60% the speed of B200s?
- fooblaster 9mo agoI wouldn't trust any benchmarks on the vendors site. Microsoft went down this path for years with FPGAs and wrote off the entire effort.
- imtringued 9mo agoI feel like your entire comment is a self contradicting mess. You say FPGAs won't get dedicated logic for ML, then you say they did. Why does it matter whether the matrix multiplication units inside the AI Engine are a systolic array or not? The multipliers support 512 bit inputs which means 4x8 times 8x4 for bfloat16 with one multiplication per cycle and bigger multiplications with smaller data types. Since it is a VLIW processor, it is much easier to achieve full utilisation of the matrix multiplication units, because you can run loads, stores and process tiles all simultaneously in the same cycle. The only thing that might be a challenge is arranging the communication between the AI Engines, but even that should be blatantly obvious. If you are doing matrix multiplication, you should be using the entire array in exactly the pattern you think they should be using internally. Who knows, maybe there is a way to implement flash attention like that too.
- fooblaster 9mo agoThe versal stuff isn't really an FPGA anymore. The chips have PL on them, but many don't. The consumer NPUs from AMD are the same versal aie cores with no PL. They just aren't configurable blocks in fabric anymore and don't have the same programming model. So I'm not contradicting myself here. That being said, versal aie for ml has been a terrible failure. The reasons for why are complicated. One reason is because the memory hierarchy for SRAM is not a unified pool. It's partitioned into tiles and can't be accessed by all cores. additionally, access of this SRAM is only via dma engines and not directly from the cores. Thirdly, the datapaths for feeding the VLIW cores are statically set, and require a software configuration to change at runtime which is slow. Programming this thing makes the cell processor look like a cakewalk. You gotta program dma engines, you program hundreds of VLIW cores, you need to explicitly setup on chip network fabric. I could go on. Anyway, my point is FPGAs aren't getting ML slices. Some FPGAs do have a completely separate thing that can do ML, but what is shipped is terrible. Hopefully that makes sense.
- teleforce 9mo ago>Those have failed by being impossible to program. I think you spoke too soon about their failure, sooner they will be much easier to program [1]. Interestingly, Nvidia GPU now is also moving to tile-based GPU programming model that targets portability for NVIDIA Tensor Cores [2]. Recently there're discussions on the topic at HN [3]. [1] Developing a BLAS Library for the AMD AI Engine [pdf]: https://uni.tlaan.nl/thesis/msc_thesis_tristan_laan_aieblas.pdf https://uni.tlaan.nl/thesis/msc_thesis_tristan_laan_aieblas.... [2] NVIDIA CUDA Tile: https://developer.nvidia.com/cuda/tile https://developer.nvidia.com/cuda/tile [3]CUDA Tile Open Sourced (103 comments): https://news.ycombinator.com/item?id=46330732 https://news.ycombinator.com/item?id=46330732
- fooblaster 9mo agoThe amd npu and versal ML tiles (same underlying architecture) have been an complete failure. Dynamic programming models like cu tile do not work on them at all, be cause they require an entirely static graph to function. AMD is going to walk away from their NPU architecture and unify around their GPU IP on inference products in the future.