6 ms·
I've been wondering when we will see general purpose consumer FPGAs, and eventually ASICs, for inference. This reminds me of bitcoin mining. Bitcoin mining star
by mrinterweb 9mo ago
I've been wondering when we will see general purpose consumer FPGAs, and eventually ASICs, for inference. This reminds me of bitcoin mining. Bitcoin mining started with GPUs. I think I remember a brief FPGA period that transitioned to ASIC. My limited understanding of Google's tensor processing unit chips are that they are effectively a transformer ASIC. That's likely a wild over-simplification of Google's TPU, but Gemini is proof that GPUs are not needed for inference.
I suspect GPU inference will come to an end soon, as it will likely be wildly inefficient by comparison to purpose built transformer chips. All those Nvidia GPU-based servers may become obsolete should transformer ASICs become mainstream. GPU bitcoin mining is just an absolute waste of money (cost of electricity) now. I believe the same will be true for GPU-based inference soon. The hundreds of billions of dollars being invested on GPU-based inference seems like an extremely risky bet that ASIC transformers won't happen, although Google has already widely deployed their own TPUs.
- tucnak 9mo agoIt all comes down to memory and fabric bandwidth. For example, the state of the art developer -friendly (PCIe 5.0) FPGA platform is Alveo V80 which rocks four 200G NIC's. Basically, Alveo currently occupies this niche where it's the only platform on the market to allow programmable in-network compute. However, what's available in terms of bandwidth—lags behind even pathetic platforms like Bluefield. Those in the know are aware of what challenges are there to actually saturate it for inference in practical designs. I think, Xilinx is super well-positioned here, but without some solid hard IP it's still a far cry from purpose silicon.
- mrinterweb 9mo agoAs far as I understand all the inference purpose-build silicon out there is not being sold to competitors and kept in-house. Google's TPU, Amazon's Inferentia (horrible name), Microsoft's Maia, Meta's MTIA. It seems that custom inference silicon is a huge part of the AI game. I doubt GPU-based inference will be relevant/competitive soon.
- almostgotcaught 9mo ago[flagged]
- mrinterweb 9mo agoSoon was wrong. I should have said it is already happening. Google Gemini already uses their own TPU chips. Nvidia just dropped $20B to buy the IP for Groq's LPU (custom silicon for inference). $20B says Nvidia sees the writing on the wall for GPU-based inference. https://www.tomshardware.com/tech-industry/semiconductors/nvidia-confirms-20-billion-groq-deal-to-bolster-ai-inference-dominance https://www.tomshardware.com/tech-industry/semiconductors/nv...
- almostgotcaught 9mo agoThere are so many people on here that are outsiders commenting way out of their depth: > Google Gemini already uses their own TPU chips Google has been using TPUs in prod for like a decade.
- nomel 9mo ago> It seems that custom inference silicon is a huge part of the AI game. Is there any public info about % inference on custom vs GPU, for these companies?
- mrinterweb 9mo agoGemini is likely the most widely used gen AI model in the world considering search, Android integration, and countless other integrations into the Google ecosystem. Gemini runs on their custom TPU chips. So I would say a large portion of inference is already using ASIC. https://cloud.google.com/tpu https://cloud.google.com/tpu
- nightshift1 9mo agoAccording to this semianalysis article, the Google/Broadcom TPU are being sold to others like Anthropic. https://newsletter.semianalysis.com/p/tpuv7-google-takes-a-swing-at-the https://newsletter.semianalysis.com/p/tpuv7-google-takes-a-s...
- fooblaster 9mo agoFPGAs will never rival gpus or TPUs for inference. The main reason is that GPUs aren't really gpus anymore. 50% of the die area or more is for fixed function matrix multiplication units and associated dedicated storage. This just isn't general purpose anymore. FPGAs cannot rival this with their configurable DSP slices. They would need dedicated systolic blocks, which they aren't getting. The closest thing is the versal ML tiles, and those are entire peoxessors, not FPGA blocks. Those have failed by being impossible to program.
- ithkuil 9mo agoTurns out that a lot of interesting computation can be expressed as a matrix multiplication.
- fooblaster 9mo agoYeah, I wouldn't have guessed it would be helping me write systemverilog.
- alanma 9mo agoyup, GBs are so much tensor core nowadays :)
- Lerc 9mo agoI think it'll get to a point with quantisation that GPUs that run them will be more FPGA like than graphics renderers. If you quantize far enough things begin to look more like gates than floating point units. At that level a FPGA wouldn't run your model, it would be one your model.
- fpgaminer 9mo ago> FPGAs will never rival gpus or TPUs for inference. The main reason is that GPUs aren't really gpus anymore. Yeah. Even for Bitcoin mining GPUs dominated FPGAs. I created the Bitcoin mining FPGA project(s), and they were only interesting for two reasons: 1) they were far more power efficient, which in the case of mining changes the equation significantly. 2) GPUs at the time had poor binary math support, which hampered their performance; whereas an FPGA is just one giant binary math machine.
- Narew 9mo agoThere was in the past. Google had Coral TPU and Intel the Neural Compute Stick (NCS). NCS is from 2018 so it's really outdated now. It was mainly oriented for edge computing so the flops was not comparable to desktop computer.
- moffkalast 9mo agoEven for edge computing neither were really even capable of keeping up with the slowest Jetson's GPU for not much less power draw.
- bee_rider 9mo agoThere are also CPU extensions like AVX512-VNNI and AVX512-BF16. Maybe the idea of communicating out to a card that holds your model will eventually go away. Inference is not too memory bandwidth hungry, right?
- liuliu 9mo agoThis is a common misunderstanding from industry observers (not industry practitioners). Each generation of (NVIDIA) GPU is an ASIC with different ISA etc. Bitcoin mining simply was not important enough (last year, only $23B Bitcoin mined in total (at $100,000 per)). There is amped incentive to implement every possible instructions useful into GPU (without worrying about backward compatibility, thanks to PTX). ASIC transformers won't happen (defined as a chip with single instruction to do sdpa from anything that is not broadly marketed as GPU, and won't have annualized sale more than $3B). Mark my word. I am happy to take a bet on longbets.org with anyone on this for $1000 and my part will go to PSF.
- dnautics 9mo agoI don't know if they'll reach $3B, but at least one company is using FPGA transformers (that perform well) to get revenue in before going to ASIC transformers: https://www.positron.ai/ https://www.positron.ai/
- zhemao 9mo agoTPUs aren't transformer ASICs. The Ironwood TPU that Gemini was trained on was designed before LLMs became popular with ChatGPT's release. The architecture was general enough that it ended up being efficient for LLM training. A special-purpose transformer inference ASIC would be like Etched's Sohu chip.
- mrinterweb 9mo ago> TPUs aren't transformer ASICs. https://cloud.google.com/tpu https://cloud.google.com/tpu > A TPU is an application-specific integrated circuit (ASIC) designed by Google for neural networks.
- seamossfet 9mo agoThe only time FPGAs / ASICS are better is if there's gains we can make by innovating on the hardware architecture itself. That's pretty hard to do considering GPUs are already heavily optimized for this use case.