3 ms·
I don't think this is correct. For inference, the bottleneck is memory bandwidth, so if you can hook up an FPGA with better memory, it has an outside shot at b
by dnautics 10mo ago
I don't think this is correct. For inference, the bottleneck is memory bandwidth, so if you can hook up an FPGA with better memory, it has an outside shot at beating GPUs, at least in the short term.
I mean, I have worked with FPGAs that outperform H200s in Llama3-class models a while and a half ago.
- fooblaster 10mo agoShow me a single FPGA that can outperform a B200 at matrix multiplication (or even come close) at any usable precision. B200 can do 10 peta ops at fp8, theoretically. I do agree memory bandwidth is also a problem for most FPGA setups, but xilinx ships HBM with some skus and they are not competitive at inference as far as I know.
- checker659 10mo agoSaid GPUs spend half the time just waiting for memory.
- fooblaster 10mo agoYep, but they are still 50x faster than any fpga.
- dnautics 10mo agoprobably not B200 level but better than you might expect: https://www.positron.ai/ https://www.positron.ai/ i believe a B200 is ~3x the H200 at llama-3, so that puts the FPGAs at around 60% the speed of B200s?
- fooblaster 10mo agoI wouldn't trust any benchmarks on the vendors site. Microsoft went down this path for years with FPGAs and wrote off the entire effort.
- dnautics 10mo agook? i worked on those devices, those numbers are real. theres a reason why they compare to h200 and not b200 > I have worked with FPGAs that outperform H200s in Llama3-class models a while and a half ago
- fooblaster 10mo agoI'd like to know more. I expect these systems are 8xvh1782. Is that true? What's the theoretical math throughput - my expectation is that it isn't very high per chip. How is performance in the prefill stage when inference is actually math limited?
- dnautics 10mo agoi was a software guy, sorry, but those token rates are correct and what was flowing through my software. i believe there was a special deal on super special fpgas. there were dsps involved.