4 ms·

It looks like this is a quantization method to flatten matrices for vector addition. Can anyone explain how this could allow LLMs to reach current benchmarks wi

• wrsh07 2y ago
  The article mentions two things. Quantization (which everyone is already doing). There's bitnet, or you can Google 1 bit quantization (it's not really 1 bit, it's ternary as the article says). Once you do that, a dot product is just addition/subtraction. Matrix multiplications are just dot products, so you've removed multiplication. Then they built custom hardware that presumably only does that operation and doesn't use much electricity Since this is just "more aggressive quantization" it's not too surprising that it reaches similar performance to other quantized models. The network should structurally look the same as any other llm, it's still using transformers etc (afaiu)
• smaddox 2y ago
  Because existing LLMs store no more than 2bits of knowledge per parameter, despite having many more bits of precision: https://arxiv.org/abs/2404.05405 https://arxiv.org/abs/2404.05405
• airspresso 2y ago
  Very sceptical of the claims that they don't lose any performance. Sounds like wishful thinking without enough effort put into measuring the performance loss that has to be there due to the heavy quantization. They even dropped computing the entire matrix addition, only focusing on some parts of it. If the benchmarks used don't show a quality drop, then that's because those benchmarks are not able to properly measure said quality drop. (edit: typo)