5 ms·

Show HN: A tiny LLM running at 21,000 tok/s on a $250 FPGA (Live Demo)

79 points by mikeayles 2mo ago

mikeayles 2mo ago
I started this about 10 weeks ago when the Taalas chatjimmy demo first did the rounds, Llama 8B baked into custom silicon, 17k tok/s for a single user. Their whole thesis is that inference is bound by reading the weights, so stop fetching them from far away. I wanted to see how far that idea stretches on a 'consumer hardware': every weight resident in SRAM, zero DRAM My chip only gives you ~3 MB to live in, so the model is 3.16M parameters and the tokens are characters, but the physics is the same bet. Only difference is their model is actually good! The annoying part: I wanted this live last week, which would have landed the same day as the AMD acquisition news, on AMD's own silicon no less. Life got in the way, so here it is a week late instead.
- antonvs 2mo ago
  IPO when? Seriously, great stuff!
haeseong 2mo ago
I didn't expect the 2,000 connection sweep to stay flat, since all of them are sharing one stream. What does per user latency look like at that end of the sweep?
- mikeayles 2mo ago
  You're correct, the flat line is aggregate only. the fabric is saturated from a few dozen active clients onward, so extra connections can't buy throughput, they just queue. per-user p50/p95 across that same sweep: 17ms/30ms solo, 450ms/545ms at 100, ~2s/2.4s at 500, 3.8s/4.4s at 1000, 6.3s/9.4s at 2000. zero errors or drops at every stage. It degrades as a well-behaved queue, not a cliff, but nobody would call 6s at the top end snappy. The benchmark sweep is 2,000 concurrent active requesters hammering it constantly, real traffic is mostly lurkers, which cost a file descriptor and nothing else. the interactive feel actually gives out earlier than the queue math. The speculative-typing UI wants sub-second replies, and that budget blows around 100–150 simultaneous typists. I've been logging the stats since it went live, unfortunately it didn't hit FP. Peak was 10 concurrent connections (13 uniques in the busiest half hour), ~580 requests and ~37k tokens served, and at no point did two people actually have an inference in flight at the same moment which would have been the real test for the queue, every visitor got the fabric to themselves, p50 ~23ms. so the 2,000-conn drill was not stressed today. the one blemish: a single window with p95 ~57s, which lines up with the model-rotation FPGA reconfigure rather than load. A request that arrives mid-reflash waits out the ~25s swap. if this thread sends 50× more people, the queue math above says it holds. I need to discard the requests that overlap the model changeover for a truer result.
  - useiris 2mo ago
    does the reflash actually stall every live connection, or just the ones whose request lands during that window? if the whole board goes dark for the full ~25s while any request is queued behind it, you could probably hide most of that behind partial reconfiguration, reflashing only the region holding the model weights while the sequencer and I/O logic on the rest of the fabric stay live and keep draining the queue. that's obviously a much bigger lift than what you've built here, but it would turn a hard stop into something closer to a brief latency bump for whoever's unlucky enough to hit it, rather than a shared 25s wall for everyone behind them in line.
    - deleted 2mo ago
      [deleted]
    - mikeayles 2mo ago
      Good question. short version: it's a shared wall, not a per-request bump. one KV260, one model on the fabric at a time, so the rotation is global. but the ~25s is mostly software, not fabric-dark time. the PL reconfigure is ~1-3s via fpgautil, the rest is orchestration (daemon teardown, streaming the weights back into URAM, clock change, coherence check, chat-server reconnect). DFX keeping the sequencer + I/O live only recovers that 1-3s slice, and it fights the design, since the weights are smeared across every URAM column while a reconfigurable partition wants a contiguous pblock. The swap needs a fresh bitstream at all only because four little ROMs are baked at synthesis: layernorm gammas, the activation-quant scale, two dequant tables. the embeddings already ride the boot-streamed weight image. make those four boot-loadable through the same path and a same-arch model hot-swaps with no reconfigure. There is basically zero margin at the moment to implement this. which is a decent excuse for a 24h telemetry update. Peak 46 concurrent connections monday afternoon, 61 uniques in the busiest half hour, ~6,600 requests / ~400k tokens total. At no instant did more than 2 inferences overlap (although it came close once). the fabric finishes a ~100-char reply in ~2ms of compute (end-to-end round-trip is ~20-60ms once you add network + server), so even with 40+ people connected the queue sits empty ~100% of the time. Worst 30-second window of the whole day was p95 59.9ms, and the heaviest burst (~200 requests in 5 minutes from one enthusiastic visitor) didn't even produce it. still ticking at ~37 connections this morning, latency flat. That's the counterintuitive bit about a 20k tok/s model with short replies: the "single stream" is busy for milliseconds per request, so it timeshares across human typing speed essentially for free. the synthetic ceiling (2,000 concurrent requesters, p95 ~9.4s, zero errors) is ~40x past anything real traffic did. every actual visitor had the chip to themselves. After this is all over, I'll do a post-mortem to see how it held up. I was really hoping for FP with the amount of effort this one took compared to my other submissions, which really would have stressed it, but I guess it was just a bit too niche!
- all2 2mo ago
  For those that don't have dead comments showing in their HN UI, Mike has a response adjacent to this one [0]. [0] https://news.ycombinator.com/item?id=49244312 https://news.ycombinator.com/item?id=49244312 @dang, the creator of this idea is having his comments killed off for some reason.
threadsnoop 2mo ago
[dead]
stevefan1999 2mo ago
But the problem is not that your model is fast. Sure, you can go ASIC and go even faster, but the thing around GPU is that they scale well for both training and inference, and the technical floor is low. The level to get into FPGA design is insanely high, you've got to read timing diagrams, you need to know combinatorial and sequential logics and good sense of boolean algebra, you need to have an asynchronous signal based mindset which is vastly different from CPU/GPU, you need to know netlist and you need to endure the time it takes for the EDA to finish generating it. Yosys is still years behind Xilinx There is a reason GPUs are called accelerators; it sacrifices and does not try to really specialize on one particular thing, except high parallel dataflow and branch-free calculation. Otherwise we will all be using DSPs
- cgyvbunji 2mo ago
  FPGAs are not power efficient at all vs GPUs and ASICs anyway, which is going to be especially true when they are fully saturated by LLM inference.
  - stevefan1999 2mo ago
    That said, FPGA do provide a middle ground, but using it for speed and power efficient is not a forte, and the true value exactly comes from this focus alone: it allows you do emulate systhesis and verify that your logic is correct before you do full ASIC tapeout, e.g. building softcores for CPU validation Anything else is added and unintentional benefits.
    - deleted 2mo ago
      [deleted]
  - adrian_b 2mo ago
    Nothing can be as power efficient as an ASIC, which is designed for a specific purpose, instead of being a programmable device intended to be suitable for a large class of applications. A GPU is much more efficient than an FPGA for what a GPU does. On the other hand for applications for which the set of primitive operations implemented in hardware by a GPU is not a good fit, an FPGA can be much more power efficient than a GPU. For applications that involve a massive amount of computations with FP32, FP16 or BF16 numbers, for which GPUs have special hardware execution units, i.e. for training and for inference with non-quantized models, there is no chance for an FPGA to be more efficient. If the GPU is recent enough to have good support for more heavily quantized data types, e.g. INT8, FP8, NVFP4 etc. an FPGA also does not have chances to be competitive. An FPGA could be more efficient than a GPU if either it is some special AI-oriented FPGA, which instead of having traditional arithmetic units oriented for DSP applications, has execution units implementing the quantized data types popular in ML/AI, or if it implements inference using some new not yet standardized data type, for which GPUs do not have dedicated support yet.
peter_d_sherman 2mo ago
Ignore the naysayers! Any article, even the really good ones on HN, while they get positive comments, for whatever reason, always get a lot of negative ones, too... That is, the negative comments are absolutely unavoidable, even for people accomplishing great things! I personally think that what you've done is brilliant, absolutely brilliant! I can't wait to see more in this space... Brilliant, absolutely brilliant!
- M4R5H4LL 2mo ago
  I am also very cautious with people who tell me something impossible when I can trust my engineering skills and get a good sense that there is potentially a good outcome. In my experience, it simply means they don’t know how to do it, or are frustrated they couldn’t do it themselves and get into the spotlight.
- RetroTechie 2mo ago
  Very nice indeed. Model weights in RAM blocks distributed all over a big FPGA: should be super helpful at minimizing RAM bandwidth bottlenecks. To say nothing of latency. But model(s) implemented are clearly too small to be useful as a 'chat partner'. Tried a couple of sentences - replies is just some gibberish coming out. This really needs a bigger FPGA, or some other application(s) where a tiny LLM does actually useful work. Barring that, generated tokens/sec is kind of a meaningless measure imho.
  - imtringued 2mo ago
    Obviously at 20k tokens per second your primary goal would be some sort of time series model running at 20KHz for processing sensor data and I'd say the model might even be too big for that. You could probably run a bunch of sensors at once.
dahnhiller 2mo ago
Hi everyone, I’m a friend of Mike’s; he’s having issues replying to the post at the moment, but hopes to post a thorough reply to the comments as soon as possible
- mikeayles 2mo ago
  Just a quick test to see if i'm back
serf 2mo ago
conceptually it's a cool idea. practically the results seem about as coherent as import random; print(random.choice(list(my_dict))) ..but way slower is there a practical use to a model this small?
- mikeayles 2mo ago
  I 100% agree the model is basically useless. that was never the deliverable. the artifact here is the inference engine, not the model living in it. 3.16M params at character level is just what fits in ~3MB of on-chip SRAM. Plus I treated it as a good learning experience to get better with FPGA's but also system design.
mrheosuper 2mo ago
[dead]
tomex 2mo ago
Crazy how little traction this kind of project gets on here. Posts about squeezing a 1+T model to seconds/token and you have this wave of optimism like "It's the effort that counts! We'll get there!". Sure, this particular project isn't really scalable in the same sense (PL fabric/use what ya got/cost/power) but IMO it's conceptually a brilliant thing to showcase comparatively. I have a strange feeling a decent chunk of people dismissing this project are the same who spent small fortunes on hobby llm inference setups/investments and see this as a useless exercise. Meanwhile dozens of $$$M startups in the CIM/analog compute/etc have been R&D'ing for years now that will make this same outcome a reality before we know it (crazy inference speeds on usable models within local reach). Anyways kudos to OP and really enjoyed the documentation and findings of this!
- mikeayles 2mo ago
  Appreciated. and yeah, agreed the interesting comparison isn't "is this scalable as-is" (it isn't, PL fabric, cost, power), it's that the CIM / analog-compute startups you mention are chasing exactly this endpoint with real money and years of R&D, the bar for making something useful is brutally high. However, if no-one made anything that was useless on the same thesis, a lot of these concepts would have never got off the ground. I would hazard a guess that people like taalas would have started with a (much much bigger) fpga to validate whether the approach was possible before committing to designing a chip big enough to fit an 8B model in it. I just nerd sniped myself... VP1902 could fit around a 500m model in, whereas a cadence protium rack of them could squeeze in a ~6B at 8bit, or a ~13B at 4bit. So accounting for the headroom of distributed compute, Llama 3.1 8B at 4bit. I don't want to even estimate how long synthesis and place and route would take on that!
variadix 2mo ago
Really cool project. I wonder what the future of LLM inference will look like. The Talaas demo is promising, but using an ASIC with weights in ROM means you can’t update the model (weights or architecture) without replacing the entire chip. SRAM isn’t dense enough to store model weights, but DRAM has bandwidth issues unless you use HBM which is expensive. Maybe novel memory technologies are the future (there are a number of emerging technologies in R&D), but they likely require breakthroughs to become commercially viable. Systolic arrays could work, one could imagine architectures where routing (architecture) is fixed but weights are programmable, or architectures where the weights and routing are programmable but the compute units are fixed function (coarse grained architecturally reprogrammable), or maybe the weights are in ROM but can be hot swapped easily with into fixed compute elements. Definitely an interesting and emerging field.
anitil 2mo ago
This is so cool, and having a live demo is just chefs kiss
gbxk 2mo ago
Amazing project, I love it. How about using these Cactus models? Would it make sense for you to collab with those guys (1) for you to design a cheap but improved, commercialisable version of your $250 chip and (2) for them to tailor their runtime and quantizations to such FPGA hardware? https://github.com/cactus-compute/cactus https://github.com/cactus-compute/cactus Also have you thought about using a Alveo V80? Still not crazy expensive and could fit bigger models with same approach

79 points by mikeayles 2mo ago

I started this about 10 weeks ago when the Taalas chatjimmy demo first did the rounds, Llama 8B baked into custom silicon, 17k tok/s for a single user. Their whole thesis is that inference is bound by reading the weights, so stop fetching them from far away. I wanted to see how far that idea stretches on a 'consumer hardware': every weight resident in SRAM, zero DRAM My chip only gives you ~3 MB to live in, so the model is 3.16M parameters and the tokens are characters, but the physics is the same bet. Only difference is their model is actually good! The annoying part: I wanted this live last week, which would have landed the same day as the AMD acquisition news, on AMD's own silicon no less. Life got in the way, so here it is a week late instead.

antonvs 2mo ago
IPO when? Seriously, great stuff!

haeseong 2mo ago

I didn't expect the 2,000 connection sweep to stay flat, since all of them are sharing one stream. What does per user latency look like at that end of the sweep?

mikeayles 2mo ago
You're correct, the flat line is aggregate only. the fabric is saturated from a few dozen active clients onward, so extra connections can't buy throughput, they just queue. per-user p50/p95 across that same sweep: 17ms/30ms solo, 450ms/545ms at 100, ~2s/2.4s at 500, 3.8s/4.4s at 1000, 6.3s/9.4s at 2000. zero errors or drops at every stage. It degrades as a well-behaved queue, not a cliff, but nobody would call 6s at the top end snappy. The benchmark sweep is 2,000 concurrent active requesters hammering it constantly, real traffic is mostly lurkers, which cost a file descriptor and nothing else. the interactive feel actually gives out earlier than the queue math. The speculative-typing UI wants sub-second replies, and that budget blows around 100–150 simultaneous typists. I've been logging the stats since it went live, unfortunately it didn't hit FP. Peak was 10 concurrent connections (13 uniques in the busiest half hour), ~580 requests and ~37k tokens served, and at no point did two people actually have an inference in flight at the same moment which would have been the real test for the queue, every visitor got the fabric to themselves, p50 ~23ms. so the 2,000-conn drill was not stressed today. the one blemish: a single window with p95 ~57s, which lines up with the model-rotation FPGA reconfigure rather than load. A request that arrives mid-reflash waits out the ~25s swap. if this thread sends 50× more people, the queue math above says it holds. I need to discard the requests that overlap the model changeover for a truer result.
- useiris 2mo ago
  does the reflash actually stall every live connection, or just the ones whose request lands during that window? if the whole board goes dark for the full ~25s while any request is queued behind it, you could probably hide most of that behind partial reconfiguration, reflashing only the region holding the model weights while the sequencer and I/O logic on the rest of the fabric stay live and keep draining the queue. that's obviously a much bigger lift than what you've built here, but it would turn a hard stop into something closer to a brief latency bump for whoever's unlucky enough to hit it, rather than a shared 25s wall for everyone behind them in line.
  - deleted 2mo ago
    [deleted]
  - mikeayles 2mo ago
    Good question. short version: it's a shared wall, not a per-request bump. one KV260, one model on the fabric at a time, so the rotation is global. but the ~25s is mostly software, not fabric-dark time. the PL reconfigure is ~1-3s via fpgautil, the rest is orchestration (daemon teardown, streaming the weights back into URAM, clock change, coherence check, chat-server reconnect). DFX keeping the sequencer + I/O live only recovers that 1-3s slice, and it fights the design, since the weights are smeared across every URAM column while a reconfigurable partition wants a contiguous pblock. The swap needs a fresh bitstream at all only because four little ROMs are baked at synthesis: layernorm gammas, the activation-quant scale, two dequant tables. the embeddings already ride the boot-streamed weight image. make those four boot-loadable through the same path and a same-arch model hot-swaps with no reconfigure. There is basically zero margin at the moment to implement this. which is a decent excuse for a 24h telemetry update. Peak 46 concurrent connections monday afternoon, 61 uniques in the busiest half hour, ~6,600 requests / ~400k tokens total. At no instant did more than 2 inferences overlap (although it came close once). the fabric finishes a ~100-char reply in ~2ms of compute (end-to-end round-trip is ~20-60ms once you add network + server), so even with 40+ people connected the queue sits empty ~100% of the time. Worst 30-second window of the whole day was p95 59.9ms, and the heaviest burst (~200 requests in 5 minutes from one enthusiastic visitor) didn't even produce it. still ticking at ~37 connections this morning, latency flat. That's the counterintuitive bit about a 20k tok/s model with short replies: the "single stream" is busy for milliseconds per request, so it timeshares across human typing speed essentially for free. the synthetic ceiling (2,000 concurrent requesters, p95 ~9.4s, zero errors) is ~40x past anything real traffic did. every actual visitor had the chip to themselves. After this is all over, I'll do a post-mortem to see how it held up. I was really hoping for FP with the amount of effort this one took compared to my other submissions, which really would have stressed it, but I guess it was just a bit too niche!
all2 2mo ago
For those that don't have dead comments showing in their HN UI, Mike has a response adjacent to this one [0]. [0] https://news.ycombinator.com/item?id=49244312 https://news.ycombinator.com/item?id=49244312 @dang, the creator of this idea is having his comments killed off for some reason.

threadsnoop 2mo ago

[dead]

stevefan1999 2mo ago

But the problem is not that your model is fast. Sure, you can go ASIC and go even faster, but the thing around GPU is that they scale well for both training and inference, and the technical floor is low. The level to get into FPGA design is insanely high, you've got to read timing diagrams, you need to know combinatorial and sequential logics and good sense of boolean algebra, you need to have an asynchronous signal based mindset which is vastly different from CPU/GPU, you need to know netlist and you need to endure the time it takes for the EDA to finish generating it. Yosys is still years behind Xilinx There is a reason GPUs are called accelerators; it sacrifices and does not try to really specialize on one particular thing, except high parallel dataflow and branch-free calculation. Otherwise we will all be using DSPs

cgyvbunji 2mo ago
FPGAs are not power efficient at all vs GPUs and ASICs anyway, which is going to be especially true when they are fully saturated by LLM inference.
- stevefan1999 2mo ago
  That said, FPGA do provide a middle ground, but using it for speed and power efficient is not a forte, and the true value exactly comes from this focus alone: it allows you do emulate systhesis and verify that your logic is correct before you do full ASIC tapeout, e.g. building softcores for CPU validation Anything else is added and unintentional benefits.
  - deleted 2mo ago
    [deleted]
- adrian_b 2mo ago
  Nothing can be as power efficient as an ASIC, which is designed for a specific purpose, instead of being a programmable device intended to be suitable for a large class of applications. A GPU is much more efficient than an FPGA for what a GPU does. On the other hand for applications for which the set of primitive operations implemented in hardware by a GPU is not a good fit, an FPGA can be much more power efficient than a GPU. For applications that involve a massive amount of computations with FP32, FP16 or BF16 numbers, for which GPUs have special hardware execution units, i.e. for training and for inference with non-quantized models, there is no chance for an FPGA to be more efficient. If the GPU is recent enough to have good support for more heavily quantized data types, e.g. INT8, FP8, NVFP4 etc. an FPGA also does not have chances to be competitive. An FPGA could be more efficient than a GPU if either it is some special AI-oriented FPGA, which instead of having traditional arithmetic units oriented for DSP applications, has execution units implementing the quantized data types popular in ML/AI, or if it implements inference using some new not yet standardized data type, for which GPUs do not have dedicated support yet.

peter_d_sherman 2mo ago

Ignore the naysayers! Any article, even the really good ones on HN, while they get positive comments, for whatever reason, always get a lot of negative ones, too... That is, the negative comments are absolutely unavoidable, even for people accomplishing great things! I personally think that what you've done is brilliant, absolutely brilliant! I can't wait to see more in this space... Brilliant, absolutely brilliant!

M4R5H4LL 2mo ago
I am also very cautious with people who tell me something impossible when I can trust my engineering skills and get a good sense that there is potentially a good outcome. In my experience, it simply means they don’t know how to do it, or are frustrated they couldn’t do it themselves and get into the spotlight.
RetroTechie 2mo ago
Very nice indeed. Model weights in RAM blocks distributed all over a big FPGA: should be super helpful at minimizing RAM bandwidth bottlenecks. To say nothing of latency. But model(s) implemented are clearly too small to be useful as a 'chat partner'. Tried a couple of sentences - replies is just some gibberish coming out. This really needs a bigger FPGA, or some other application(s) where a tiny LLM does actually useful work. Barring that, generated tokens/sec is kind of a meaningless measure imho.
- imtringued 2mo ago
  Obviously at 20k tokens per second your primary goal would be some sort of time series model running at 20KHz for processing sensor data and I'd say the model might even be too big for that. You could probably run a bunch of sensors at once.

dahnhiller 2mo ago

Hi everyone, I’m a friend of Mike’s; he’s having issues replying to the post at the moment, but hopes to post a thorough reply to the comments as soon as possible

mikeayles 2mo ago
Just a quick test to see if i'm back

serf 2mo ago

conceptually it's a cool idea. practically the results seem about as coherent as import random; print(random.choice(list(my_dict))) ..but way slower is there a practical use to a model this small?

mikeayles 2mo ago
I 100% agree the model is basically useless. that was never the deliverable. the artifact here is the inference engine, not the model living in it. 3.16M params at character level is just what fits in ~3MB of on-chip SRAM. Plus I treated it as a good learning experience to get better with FPGA's but also system design.

mrheosuper 2mo ago

[dead]

tomex 2mo ago

Crazy how little traction this kind of project gets on here. Posts about squeezing a 1+T model to seconds/token and you have this wave of optimism like "It's the effort that counts! We'll get there!". Sure, this particular project isn't really scalable in the same sense (PL fabric/use what ya got/cost/power) but IMO it's conceptually a brilliant thing to showcase comparatively. I have a strange feeling a decent chunk of people dismissing this project are the same who spent small fortunes on hobby llm inference setups/investments and see this as a useless exercise. Meanwhile dozens of $$$M startups in the CIM/analog compute/etc have been R&D'ing for years now that will make this same outcome a reality before we know it (crazy inference speeds on usable models within local reach). Anyways kudos to OP and really enjoyed the documentation and findings of this!

mikeayles 2mo ago
Appreciated. and yeah, agreed the interesting comparison isn't "is this scalable as-is" (it isn't, PL fabric, cost, power), it's that the CIM / analog-compute startups you mention are chasing exactly this endpoint with real money and years of R&D, the bar for making something useful is brutally high. However, if no-one made anything that was useless on the same thesis, a lot of these concepts would have never got off the ground. I would hazard a guess that people like taalas would have started with a (much much bigger) fpga to validate whether the approach was possible before committing to designing a chip big enough to fit an 8B model in it. I just nerd sniped myself... VP1902 could fit around a 500m model in, whereas a cadence protium rack of them could squeeze in a ~6B at 8bit, or a ~13B at 4bit. So accounting for the headroom of distributed compute, Llama 3.1 8B at 4bit. I don't want to even estimate how long synthesis and place and route would take on that!

variadix 2mo ago

Really cool project. I wonder what the future of LLM inference will look like. The Talaas demo is promising, but using an ASIC with weights in ROM means you can’t update the model (weights or architecture) without replacing the entire chip. SRAM isn’t dense enough to store model weights, but DRAM has bandwidth issues unless you use HBM which is expensive. Maybe novel memory technologies are the future (there are a number of emerging technologies in R&D), but they likely require breakthroughs to become commercially viable. Systolic arrays could work, one could imagine architectures where routing (architecture) is fixed but weights are programmable, or architectures where the weights and routing are programmable but the compute units are fixed function (coarse grained architecturally reprogrammable), or maybe the weights are in ROM but can be hot swapped easily with into fixed compute elements. Definitely an interesting and emerging field.

anitil 2mo ago

This is so cool, and having a live demo is just chefs kiss

gbxk 2mo ago

Amazing project, I love it. How about using these Cactus models? Would it make sense for you to collab with those guys (1) for you to design a cheap but improved, commercialisable version of your $250 chip and (2) for them to tailor their runtime and quantizations to such FPGA hardware? https://github.com/cactus-compute/cactus https://github.com/cactus-compute/cactus Also have you thought about using a Alveo V80? Still not crazy expensive and could fit bigger models with same approach