5 ms·

The big challenge when it comes to using FPGAs for deep learning is pretty simple: all of that reprogrammability comes at a performance cost. If you're doing so

The big challenge when it comes to using FPGAs for deep learning is pretty simple: all of that reprogrammability comes at a performance cost. If you're doing something highly specific that conventional GPUs are bad at, like genomics research [1] or high-frequency trading [2], the performance tradeoff is worth it. But for deep learning, GPUs and AI ASICs are highly optimized for most of these computations, and an FPGA won't offer huge performance increases. The main advantage FPGAs offer is being able to take advantage of new model optimizations much earlier than ASIC implementations could. Those proposed ternary LLMs could potentially run much faster on FPGAs, because the hardware could be optimized for exclusively ternary ops. [3] Not to toot my own horn, but I wrote up a blog post recently about building practical FPGA acceleration and which applications are best suited for it: https://www.zach.be/p/how-to-build-a-commercial-open-source https://www.zach.be/p/how-to-build-a-commercial-open-source [1] https://aws.amazon.com/solutions/case-studies/munich-leukemia-lab/ https://aws.amazon.com/solutions/case-studies/munich-leukemi... [2] https://careers.imc.com/us/en/blogarticle/how-are-fpgas-used-in-trading https://careers.imc.com/us/en/blogarticle/how-are-fpgas-used... [3] https://arxiv.org/abs/2402.17764 https://arxiv.org/abs/2402.17764

jacobgorm 2y ago
I was part of a startup that did ternary CNNs on FPGA in 2017. It involved a ton of nitty gritty work and massive loss of generalit, and in the end a Raspberry Pi could solve the same problem faster and cheaper.
adultSwim 2y ago
What about FPGAs as a means to experiment in ML and hardware architectures?
kadushka 2y ago
There are two other problems with FPGAs: 1. They are hard to use (program). If you're a regular ML engineer, there will be a steep learning curve with Verilog/VHDL and the specifics of the chip you choose, especially if you want to squeeze all the performance out of it. For most researchers it's just not worth it. And for production deployment it's not worth the risk of investing into an unproven platform. Microsoft tried it many years ago to accelerate their search/whatever, and I think they abandoned it. 2. Cost. High performance FPGA chips are expensive. Like A100 to H100 price range. Very few people would be willing to spend this much to accelerate their DL models unless the speedup is > 2x compared to GPUs.
IX-103 2y ago
FPGAs are also reasonably good at breadboarding modules to be added to ASICs. You scale down the timing and you can run the same HDL and perform software integration at the same time as the HDL is optimized. Much cheaper and faster than gate level simulation.
sitkack 2y ago
Are you trying to scare people away from FPGAs? GPUs aren't actually that _good_ at deep learning, but they are in the right place at the right time. You can rent high end FPGAs on AWS, https://github.com/aws/aws-fpga https://github.com/aws/aws-fpga there is no better time to get into FPGAs. On the low end there is the excellent https://hackaday.com/2019/01/14/ulx3s-an-open-source-lattice-ecp5-fpga-pcb/ https://hackaday.com/2019/01/14/ulx3s-an-open-source-lattice... Modern FPGA platforms like Xilinx Alveo have 35TB/s of SRAM bandwidth and 460GB/s of HBM bandwidth. https://www.xilinx.com/products/boards-and-kits/alveo/u55c.html#specifications https://www.xilinx.com/products/boards-and-kits/alveo/u55c.h...
- weinzierl 2y ago
  If I remember correctly about 80% of a modern FPGA's silicon is is used for connections. FPGA have their uses and very often a big part in them is the Field Programmability. If that is not required, there is no good reason another solution (ASIC, GPU, etc.) couldn't beat the FPGA in theory. Now, in practice there are some niches, where this is not absolutely true, but I agree with GP that I see challenges for deep learning.
  - adrian_b 2y ago
    An ASIC will always have better performance than an FPGA, but it will have an acceptable cost only if it is produced in a large enough number. You will always want an ASIC, but only seldom you will able to afford it. So the decision of ASIC vs. FPGA is trivial, it is always based on the estimated price of the ASIC, based on the number of ASICs that would be needed. The decision between off-the-shelf components, i.e. GPUs and FPGAs, is done based on performance per dollar and performance per W and it depends very strongly on the intended application. If the application must compute many operations with bigger numbers, e.g. FP32 or FP16, then it is unlikely that an FPGA can compete with a GPU. When arithmetic computations do not form the bulk of an algorithm, then an FPGA may be competitive, but a detailed analysis must be made for any specific application.
- tehsauce 2y ago
  500GB/s is going to limit it to at best 1/4 the DL performance of an nvidia gpu. I’m not sure what the floating point perf of these FPGAs are but I imagine that also might set a fundamental performance limit at a small fraction of a GPU.
weinzierl 2y ago
Every couple of years I revisit the FPGA topic, eager to build something exciting. I always end up with a ton of research, where I learn a lot but ultimately shy away from building something. This is because I cannot find a project that is doable and affordable for a hobbyist but at the same time requires an FPGA in some sense. To put it bluntly: I can blink a LED for a fiver with a micro instead of spending hundreds for an FPGA. So, assuming I am reasonably experienced in software development and electronics and I have 1000 USD and a week to spend. What could I build that shows off the capabilities of an FPGA?
- scottapotamas 2y ago
  Reasonably experienced and 'a week' can mean vastly different things... It's certainly easier to keep the cost down with longer time-frames. For a focus on electronics rather than implementing some kind of toy 'algorithm accelerator', I find low-hanging/interesting projects where the combination of requirements exceed a micro's peripheral capabilities - i.e. multiple input/output/processing tasks which could be performed on a micro individually, but adding synchronisation or latency requirements makes it rather non-trivial. - Very wide/parallel input/output tasks: ADC/DACs for higher samplerate/bitdepth/channel count than typically accessible with even high-end micros - Implementing unique/specialised protocols which would have required bit-banging, abuse of timer/other peripherals on a micro (i.e. interesting things people achieve with PIO blocks on RP2040 etc) - Signal processing: digital filters and control systems are great because you can see/hear/interact with the output which can help build a sense of achievement. When starting out, it's also less overwhelming to start with smaller parts and allocate the budget to the rest of the electronics. They're still incredibly capable and won't seem as under-utilised. Some random project ideas: - Driving large frame-buffers to display(s) or large sets of LED matrices at high frame rate - https://gregdavill.com/posts/d20/ https://gregdavill.com/posts/d20/ - Realtime audio filters - the Eurorack community might have some inspiration. - Multi-channel synchonous detection, lock-in amplifiers, distributed timing reference/control, - Find a sensing application that's interesting and then take it to the logical extreme - arrays of photo/hall-effect sensors sampled at high speed and displayed, accelerometers/IMU sensor fusion - Laser galvanometers and piezo actuators are getting more accessible - Small but precise/fast motion stages for positioning or sensing might present a good combination of input, output, filtering and control systems. - With more time/experience you could branch into more interesting (IMO) areas like RF or imaging systems. With more info about your interest areas I can give more specific suggestions.
imtringued 2y ago
Um, no? The actual problem is that most FPGAs already have DPUs for machine learning integrated on them. Some Xilinx FPGAs have 400 "AI Engines" which provide significantly more compute than the programmable logic, the almost 2000 DSP slices or the ARM cores. This means that the problem with FPGAs is primarily lack of SRAM and limited memory bandwidth. https://www.xilinx.com/products/boards-and-kits/vck190.html https://www.xilinx.com/products/boards-and-kits/vck190.html

jacobgorm 2y ago

I was part of a startup that did ternary CNNs on FPGA in 2017. It involved a ton of nitty gritty work and massive loss of generalit, and in the end a Raspberry Pi could solve the same problem faster and cheaper.

adultSwim 2y ago

What about FPGAs as a means to experiment in ML and hardware architectures?

kadushka 2y ago

There are two other problems with FPGAs: 1. They are hard to use (program). If you're a regular ML engineer, there will be a steep learning curve with Verilog/VHDL and the specifics of the chip you choose, especially if you want to squeeze all the performance out of it. For most researchers it's just not worth it. And for production deployment it's not worth the risk of investing into an unproven platform. Microsoft tried it many years ago to accelerate their search/whatever, and I think they abandoned it. 2. Cost. High performance FPGA chips are expensive. Like A100 to H100 price range. Very few people would be willing to spend this much to accelerate their DL models unless the speedup is > 2x compared to GPUs.

IX-103 2y ago

FPGAs are also reasonably good at breadboarding modules to be added to ASICs. You scale down the timing and you can run the same HDL and perform software integration at the same time as the HDL is optimized. Much cheaper and faster than gate level simulation.

sitkack 2y ago

Are you trying to scare people away from FPGAs? GPUs aren't actually that _good_ at deep learning, but they are in the right place at the right time. You can rent high end FPGAs on AWS, https://github.com/aws/aws-fpga https://github.com/aws/aws-fpga there is no better time to get into FPGAs. On the low end there is the excellent https://hackaday.com/2019/01/14/ulx3s-an-open-source-lattice-ecp5-fpga-pcb/ https://hackaday.com/2019/01/14/ulx3s-an-open-source-lattice... Modern FPGA platforms like Xilinx Alveo have 35TB/s of SRAM bandwidth and 460GB/s of HBM bandwidth. https://www.xilinx.com/products/boards-and-kits/alveo/u55c.html#specifications https://www.xilinx.com/products/boards-and-kits/alveo/u55c.h...

weinzierl 2y ago
If I remember correctly about 80% of a modern FPGA's silicon is is used for connections. FPGA have their uses and very often a big part in them is the Field Programmability. If that is not required, there is no good reason another solution (ASIC, GPU, etc.) couldn't beat the FPGA in theory. Now, in practice there are some niches, where this is not absolutely true, but I agree with GP that I see challenges for deep learning.
- adrian_b 2y ago
  An ASIC will always have better performance than an FPGA, but it will have an acceptable cost only if it is produced in a large enough number. You will always want an ASIC, but only seldom you will able to afford it. So the decision of ASIC vs. FPGA is trivial, it is always based on the estimated price of the ASIC, based on the number of ASICs that would be needed. The decision between off-the-shelf components, i.e. GPUs and FPGAs, is done based on performance per dollar and performance per W and it depends very strongly on the intended application. If the application must compute many operations with bigger numbers, e.g. FP32 or FP16, then it is unlikely that an FPGA can compete with a GPU. When arithmetic computations do not form the bulk of an algorithm, then an FPGA may be competitive, but a detailed analysis must be made for any specific application.
tehsauce 2y ago
500GB/s is going to limit it to at best 1/4 the DL performance of an nvidia gpu. I’m not sure what the floating point perf of these FPGAs are but I imagine that also might set a fundamental performance limit at a small fraction of a GPU.

weinzierl 2y ago

Every couple of years I revisit the FPGA topic, eager to build something exciting. I always end up with a ton of research, where I learn a lot but ultimately shy away from building something. This is because I cannot find a project that is doable and affordable for a hobbyist but at the same time requires an FPGA in some sense. To put it bluntly: I can blink a LED for a fiver with a micro instead of spending hundreds for an FPGA. So, assuming I am reasonably experienced in software development and electronics and I have 1000 USD and a week to spend. What could I build that shows off the capabilities of an FPGA?

scottapotamas 2y ago
Reasonably experienced and 'a week' can mean vastly different things... It's certainly easier to keep the cost down with longer time-frames. For a focus on electronics rather than implementing some kind of toy 'algorithm accelerator', I find low-hanging/interesting projects where the combination of requirements exceed a micro's peripheral capabilities - i.e. multiple input/output/processing tasks which could be performed on a micro individually, but adding synchronisation or latency requirements makes it rather non-trivial. - Very wide/parallel input/output tasks: ADC/DACs for higher samplerate/bitdepth/channel count than typically accessible with even high-end micros - Implementing unique/specialised protocols which would have required bit-banging, abuse of timer/other peripherals on a micro (i.e. interesting things people achieve with PIO blocks on RP2040 etc) - Signal processing: digital filters and control systems are great because you can see/hear/interact with the output which can help build a sense of achievement. When starting out, it's also less overwhelming to start with smaller parts and allocate the budget to the rest of the electronics. They're still incredibly capable and won't seem as under-utilised. Some random project ideas: - Driving large frame-buffers to display(s) or large sets of LED matrices at high frame rate - https://gregdavill.com/posts/d20/ https://gregdavill.com/posts/d20/ - Realtime audio filters - the Eurorack community might have some inspiration. - Multi-channel synchonous detection, lock-in amplifiers, distributed timing reference/control, - Find a sensing application that's interesting and then take it to the logical extreme - arrays of photo/hall-effect sensors sampled at high speed and displayed, accelerometers/IMU sensor fusion - Laser galvanometers and piezo actuators are getting more accessible - Small but precise/fast motion stages for positioning or sensing might present a good combination of input, output, filtering and control systems. - With more time/experience you could branch into more interesting (IMO) areas like RF or imaging systems. With more info about your interest areas I can give more specific suggestions.

imtringued 2y ago

Um, no? The actual problem is that most FPGAs already have DPUs for machine learning integrated on them. Some Xilinx FPGAs have 400 "AI Engines" which provide significantly more compute than the programmable logic, the almost 2000 DSP slices or the ARM cores. This means that the problem with FPGAs is primarily lack of SRAM and limited memory bandwidth. https://www.xilinx.com/products/boards-and-kits/vck190.html https://www.xilinx.com/products/boards-and-kits/vck190.html