2 ms·
Said GPUs spend half the time just waiting for memory.
by checker659 9mo ago
Said GPUs spend half the time just waiting for memory.
- fooblaster 9mo agoYep, but they are still 50x faster than any fpga.
- dnautics 9mo agoprobably not B200 level but better than you might expect: https://www.positron.ai/ https://www.positron.ai/ i believe a B200 is ~3x the H200 at llama-3, so that puts the FPGAs at around 60% the speed of B200s?
- fooblaster 9mo agoI wouldn't trust any benchmarks on the vendors site. Microsoft went down this path for years with FPGAs and wrote off the entire effort.
- dnautics 9mo agook? i worked on those devices, those numbers are real. theres a reason why they compare to h200 and not b200 > I have worked with FPGAs that outperform H200s in Llama3-class models a while and a half ago
- fooblaster 9mo agoI'd like to know more. I expect these systems are 8xvh1782. Is that true? What's the theoretical math throughput - my expectation is that it isn't very high per chip. How is performance in the prefill stage when inference is actually math limited?
- dnautics 9mo agoi was a software guy, sorry, but those token rates are correct and what was flowing through my software. i believe there was a special deal on super special fpgas. there were dsps involved.