3 ms·

See also the blog post. https://timdettmers.com/2022/08/17/llm-int8-and-emergent-features/ https://timdettmers.com/2022/08/17/llm-int8-and-emergent-fea... I pe

See also the blog post. https://timdettmers.com/2022/08/17/llm-int8-and-emergent-features/ https://timdettmers.com/2022/08/17/llm-int8-and-emergent-fea... I personally found most interesting of this work the emergent sparse behavior starting around 6B parameters but not before. That suggest that the model is operating in a different mode at such model size or larger.

brucethemoose2 3y ago
A continuation of that trend (that, for instance, a 65B model would quantize better than a 33B one) was popular belief, but the k-quants dev recently found that LLaMA 7B-65B all quantize very similarly: https://github.com/ggerganov/llama.cpp/pull/1684 https://github.com/ggerganov/llama.cpp/pull/1684
- sroussey 3y ago
  That graph really puts it all into perspective too.
heyitsguay 3y ago
Sounds like the idea is that "outlier features" effectively give the model an inhibitory system for other features, and transformer models converge to consistent outlier channel selection per layer at around the 6.7B param mark? It's interesting, biological neural networks are known to use inhibitory systems for signal gating and sharpening/sparsification, but it's typically handled by distinct neuron types that are strictly inhibitory. I wonder if one could boost smaller transformer performance with an analogous construction - add on a small additional module constructed for channel inhibition in the rest of the model.