4 ms·
The claim about moving closer to brain-like efficiency conveniently omits how that model compares to modern LLMs. You can put together a toy LLM that is much sm
by Aurornis 2y ago
The claim about moving closer to brain-like efficiency conveniently omits how that model compares to modern LLMs. You can put together a toy LLM that is much smaller and more efficient than ChatGPT but isn’t as useful and call it “more efficient”, but that’s not useful in practice.
- yashap 2y agoThey do cover that in the article: > Although they reduced the number of operations, the researchers were able to maintain the performance of the neural network by introducing time-based computation in the training of the model. This enables the network to have a “memory” of the important information it processes, enhancing performance. This technique paid off — the researchers compared their model to Meta’s state-of-the-art algorithm called Llama, and were able to achieve the same performance, even at a scale of billions of model parameters.
- cs702 2y ago> The claim about moving closer to brain-like efficiency conveniently omits how that model compares to modern LLMs. I disagree. The authors aren't conveniently omitting anything. They show all details in a comparison against LLama models. Moreover, all evidence I've seen so far suggests that tritwise models can scale up to state-of-the-art sizes. --- PS. I'm talking about the paper, not the fluffy press release.
- juliangamble 2y agoI’ve heard some claims that to get closer to brain-like energy efficiency you’d need to use a spiking neural network https://en.wikipedia.org/wiki/Spiking_neural_network https://en.wikipedia.org/wiki/Spiking_neural_network