3 ms·

See also: https://rits.shanghai.nyu.edu/ai/karpathys-microgpt-on-fpga-loses-to-a-single-macbook-core-by-71x/ https://rits.shanghai.nyu.edu/ai/karpathys-microgp

by amelius 5mo ago

See also: https://rits.shanghai.nyu.edu/ai/karpathys-microgpt-on-fpga-loses-to-a-single-macbook-core-by-71x/ https://rits.shanghai.nyu.edu/ai/karpathys-microgpt-on-fpga-... TL;DR: The CPU implementation was 71x faster than the FPGA. Note: model has only 4192 parameters.

cyanydeez 5mo ago
yeah, then theres prompt loading too. but anyone who can fit QWEN-3.6 35B with a sustained ~30 token/s and ~100k context with cache could print money as a hardware vendor.
- wmf 5mo ago
  That just sounds like a 3090.
  - cyanydeez 5mo ago
    not at the vram sizes that control how much context to load; also, GPUs arn't as effiecient as direct inference.
    - wmf 5mo ago
      OK, B70.
- upboundspiral 5mo ago
  with llama-cpp and offloading non-active experts (from MOE architecture) to cpu RAM, you can easily run 50 tok / s QWEN-3.6 35B on 8-12 GB of VRAM. KV cache is a few GB, experts are ~3-5 GB (assuming q8 quant from Unsloth for example). You can scroll through r/localllama and find tons of people getting useable speeds out of Qwen 35B. 24 tok / second on an ancient 1080ti https://old.reddit.com/r/LocalLLaMA/comments/1tcc7h5/24_toks_from_30b_moe_models_on_an_old_gtx_1080_8/ https://old.reddit.com/r/LocalLLaMA/comments/1tcc7h5/24_toks... 100 tok / second on a 4070 https://old.reddit.com/r/LocalLLaMA/comments/1tjh7az/110_toks_with_12gb_vram_on_qwen36_35b_a3b_and_ik/ https://old.reddit.com/r/LocalLLaMA/comments/1tjh7az/110_tok...
hedgehog 5mo ago
That post is uninteresting both because they miss the point, and it's not clear a human was even involved to perceive a point to miss. Sure, with an unlimited transistor budget, power budget, and a design clocked at 4GHz fabbed on 5nm one of the best CPU design teams in the world can make a thing that is straight line faster than a one-person project running at 80MHz on a 20 year old 65nm FPGA. Any other answer would be extremely surprising. Now, there are a bunch of interesting things about this project. Seeing the example of a tiny transformer running on FPGA is informative, and that it was apparently a pretty quick project for one person + robot assistance. Probably some transferable lessons for anyone else doing robo-FPGA development. https://github.com/fguzman82/gateGPT/tree/main/ https://github.com/fguzman82/gateGPT/tree/main/

by amelius 5mo ago

cyanydeez 5mo ago

yeah, then theres prompt loading too. but anyone who can fit QWEN-3.6 35B with a sustained ~30 token/s and ~100k context with cache could print money as a hardware vendor.

wmf 5mo ago
That just sounds like a 3090.
- cyanydeez 5mo ago
  not at the vram sizes that control how much context to load; also, GPUs arn't as effiecient as direct inference.
  - wmf 5mo ago
    OK, B70.
upboundspiral 5mo ago
with llama-cpp and offloading non-active experts (from MOE architecture) to cpu RAM, you can easily run 50 tok / s QWEN-3.6 35B on 8-12 GB of VRAM. KV cache is a few GB, experts are ~3-5 GB (assuming q8 quant from Unsloth for example). You can scroll through r/localllama and find tons of people getting useable speeds out of Qwen 35B. 24 tok / second on an ancient 1080ti https://old.reddit.com/r/LocalLLaMA/comments/1tcc7h5/24_toks_from_30b_moe_models_on_an_old_gtx_1080_8/ https://old.reddit.com/r/LocalLLaMA/comments/1tcc7h5/24_toks... 100 tok / second on a 4070 https://old.reddit.com/r/LocalLLaMA/comments/1tjh7az/110_toks_with_12gb_vram_on_qwen36_35b_a3b_and_ik/ https://old.reddit.com/r/LocalLLaMA/comments/1tjh7az/110_tok...

hedgehog 5mo ago

That post is uninteresting both because they miss the point, and it's not clear a human was even involved to perceive a point to miss. Sure, with an unlimited transistor budget, power budget, and a design clocked at 4GHz fabbed on 5nm one of the best CPU design teams in the world can make a thing that is straight line faster than a one-person project running at 80MHz on a 20 year old 65nm FPGA. Any other answer would be extremely surprising. Now, there are a bunch of interesting things about this project. Seeing the example of a tiny transformer running on FPGA is informative, and that it was apparently a pretty quick project for one person + robot assistance. Probably some transferable lessons for anyone else doing robo-FPGA development. https://github.com/fguzman82/gateGPT/tree/main/ https://github.com/fguzman82/gateGPT/tree/main/