4 ms·

Paper: https://arxiv.org/abs/2406.02528 https://arxiv.org/abs/2406.02528 -- always better than a press release. Code: https://github.com/ridgerchu/matmulfreell

by cs702 2y ago

Paper: https://arxiv.org/abs/2406.02528 https://arxiv.org/abs/2406.02528 -- always better than a press release. Code: https://github.com/ridgerchu/matmulfreellm https://github.com/ridgerchu/matmulfreellm --- Like others before them, the authors train LLMs using parameters consisting of ternary digits, or trits, with values in {-1, 0, 1}. What's new is that the authors then build a custom hardware solution on an FPGA and run billion-parameter LLMs consuming only 13W, moving LLM inference closer to brain-like efficiency. Sure, it's on an FPGA, and it's only a lab experiment, but we're talking about an early proof of concept, not a commercial product. As far as I know, this is the first energy-efficient hardware implementation of tritwise LLMs. That seems like a pretty big deal to me.

Aurornis 2y ago
The claim about moving closer to brain-like efficiency conveniently omits how that model compares to modern LLMs. You can put together a toy LLM that is much smaller and more efficient than ChatGPT but isn’t as useful and call it “more efficient”, but that’s not useful in practice.
- yashap 2y ago
  They do cover that in the article: > Although they reduced the number of operations, the researchers were able to maintain the performance of the neural network by introducing time-based computation in the training of the model. This enables the network to have a “memory” of the important information it processes, enhancing performance. This technique paid off — the researchers compared their model to Meta’s state-of-the-art algorithm called Llama, and were able to achieve the same performance, even at a scale of billions of model parameters.
- cs702 2y ago
  > The claim about moving closer to brain-like efficiency conveniently omits how that model compares to modern LLMs. I disagree. The authors aren't conveniently omitting anything. They show all details in a comparison against LLama models. Moreover, all evidence I've seen so far suggests that tritwise models can scale up to state-of-the-art sizes. --- PS. I'm talking about the paper, not the fluffy press release.
  - avisser 2y ago
    I took the critique as being against OP, not the paper.
    - cs702 2y ago
      Ah, that makes more sense :-) Thanks for pointing it out! PS. I added a PS to my comment above.
- juliangamble 2y ago
  I’ve heard some claims that to get closer to brain-like energy efficiency you’d need to use a spiking neural network https://en.wikipedia.org/wiki/Spiking_neural_network https://en.wikipedia.org/wiki/Spiking_neural_network
lawlessone 2y ago
> moving LLM inference closer to brain-like efficiency. Yeah but the brain does more than predictive text.
- fuzzfactor 2y ago
  At least they're aware of what it is to be on the wrong track. According to the researchers, >all we had to do was fundamentally change how neural networks work,
larodi 2y ago
to me the most interesting part in this all is the quantization used. hardware doing LLMs is likely to be the new norm in few years anyway. with retrofitting existing hardware to use say USBs running LLM accelerators and alike...
cpeterso 2y ago
I assume the ternary weight’s memory representation requires two bits, so why do they use only three values instead of four? OTOH, I’m not sure what fourth value would useful for LLM models doing math with -1, 0, and 1. Infinity? NaN?

by cs702 2y ago

Aurornis 2y ago

The claim about moving closer to brain-like efficiency conveniently omits how that model compares to modern LLMs. You can put together a toy LLM that is much smaller and more efficient than ChatGPT but isn’t as useful and call it “more efficient”, but that’s not useful in practice.

yashap 2y ago
They do cover that in the article: > Although they reduced the number of operations, the researchers were able to maintain the performance of the neural network by introducing time-based computation in the training of the model. This enables the network to have a “memory” of the important information it processes, enhancing performance. This technique paid off — the researchers compared their model to Meta’s state-of-the-art algorithm called Llama, and were able to achieve the same performance, even at a scale of billions of model parameters.
cs702 2y ago
> The claim about moving closer to brain-like efficiency conveniently omits how that model compares to modern LLMs. I disagree. The authors aren't conveniently omitting anything. They show all details in a comparison against LLama models. Moreover, all evidence I've seen so far suggests that tritwise models can scale up to state-of-the-art sizes. --- PS. I'm talking about the paper, not the fluffy press release.
- avisser 2y ago
  I took the critique as being against OP, not the paper.
  - cs702 2y ago
    Ah, that makes more sense :-) Thanks for pointing it out! PS. I added a PS to my comment above.
juliangamble 2y ago
I’ve heard some claims that to get closer to brain-like energy efficiency you’d need to use a spiking neural network https://en.wikipedia.org/wiki/Spiking_neural_network https://en.wikipedia.org/wiki/Spiking_neural_network

lawlessone 2y ago

> moving LLM inference closer to brain-like efficiency. Yeah but the brain does more than predictive text.

fuzzfactor 2y ago
At least they're aware of what it is to be on the wrong track. According to the researchers, >all we had to do was fundamentally change how neural networks work,

larodi 2y ago

to me the most interesting part in this all is the quantization used. hardware doing LLMs is likely to be the new norm in few years anyway. with retrofitting existing hardware to use say USBs running LLM accelerators and alike...

cpeterso 2y ago

I assume the ternary weight’s memory representation requires two bits, so why do they use only three values instead of four? OTOH, I’m not sure what fourth value would useful for LLM models doing math with -1, 0, and 1. Infinity? NaN?