4 ms·
The press release is devoid of useful information, unsurprisingly. You can run an LLM under almost any energy envelope if you’re willing to wait long enough for
by Aurornis 2y ago
The press release is devoid of useful information, unsurprisingly. You can run an LLM under almost any energy envelope if you’re willing to wait long enough for the result. Total energy consumed and the time difference are the more important metrics.
The actual paper is here: https://arxiv.org/abs/2406.02528 https://arxiv.org/abs/2406.02528
The key part from the summary:
> To properly quantify the efficiency of our architecture, we build a custom hardware solution on an FPGA which exploits lightweight operations beyond what GPUs are capable of. We processed billion-parameter scale models at 13W beyond human readable throughput, moving LLMs closer to brain-like efficiency.
There is a lot of unnecessary obfuscation of the numbers going on in the abstract as well, which is unfortunate. Instead of quoting the numbers they call it “billion-parameter scale” and “beyond human readable throughout”.
- FpUser 2y agoThey did say that the answer is being produced faster than the human can read.
- boto3 2y agoyes, human average reading speed is ~250 words per minute, but to be fair, it's not a widely known stat.
- kevmo 2y ago250 seems high? I want to believe, though.
- BenjiWiebe 2y agoThat's 4 words per second.
- mensetmanusman 2y agoSpeaking is 150 wpm
- onlyrealcuzzo 2y agoIt's almost as if our brain is optimized for taking in more signals than just words on a page.
- pbnjay 2y agoYeah... They are using a single-core 13W measurement to project out. For a 64x parallelization - no mention of any overhead due to parallelization or power needs of the supporting hardware. This is a key quote for me (page 12 of the PDF): > The 1.3B parameter model, where L = 24 and d = 2048, has a projected runtime of 42ms, and a throughput of 23.8 tokens per second. e.g. 64 x 13.67W = 874 Watts to run a 1.3B model at 23.8 t/s... I'm pretty sure my phone can do way better than that! Even half that power given their assertions in the table are still overpowered for such a small model.
- shivaluminaire 2y agoWhen you multiply by 64 you also get 64 times more tokens per second!! Your math is wrong.
- shivaluminaire 2y agoWhen you multiply by 64 you also get 64 times more tokens per second!! Your math is wrong.
- pbnjay 2y agoThat's their math, the 23.8t/s is already the 64x but they didn't 64x the other stats.