3 ms·

This really just sounds absurd. How can ternary possibly encode enough information? Anyone willing to explain it like I’m a Django developer who watched half a

This really just sounds absurd. How can ternary possibly encode enough information? Anyone willing to explain it like I’m a Django developer who watched half a karpathy video?

binarist 3y ago
[dead]
Solvency 3y ago
Because by making the model larger you don't need 64bit precision floats you only need 64 discrete bits.
- gemeral 3y ago
  Do you mind pointing out where they make the model larger? The paper seems to suggest they are maintaining the same model sizes. > Recent research, such as BitNet, is paving the way for a new era of 1-bit Large Language Models (LLMs). In this work, we introduce a 1-bit LLM variant, namely BitNet b1.58, in which every single parameter (or weight) of the LLM is ternary {-1, 0, 1}. It matches the full-precision (i.e., FP16 or BF16) Transformer LLM with the same model size and training tokens in terms of both perplexity and end-task performance, while being significantly more cost-effective in terms of latency, memory, throughput, and energy consumption
barbarr 3y ago
The activations are still 8-bit, so a lot of complexity and nonlinearity is still expressible. Only the weights are 1.58-bit.
HanClinto 3y ago
On its own, each trit doesn't encode much information at all. But it's not about information at the individual level -- it's more about the shape of the network. I appreciated this comment [0] from earlier in the thread by paul_mk1: > My best guess is that it is encouraging the network to choose good underlying subnetworks to solve the problem, similar to Lottery Ticket Hypothesis. With ternary weights it is just about who connects to who (ie a graph), and not about the individual weight values anymore. For myself, I've done a lot of work with image hashing (such as pHash and dHash) -- and in those, you throw away a LOT of information, but simply by keeping the value of each region and tracking whether or not it's above or below the average (essentially, the sign), then it's astounding how robust those algorithms are. Because you don't look at the individual pixels of an image, but it's very good at capturing the impression of the overall _shape_ of the image. It's less about each individual datum, and more about the shape of the network. If you're not familiar with Lottery Ticket Hypothesis, that would be worth reading up on. [0]: https://news.ycombinator.com/item?id=39544500 https://news.ycombinator.com/item?id=39544500

This really just sounds absurd. How can ternary possibly encode enough information? Anyone willing to explain it like I’m a Django developer who watched half a karpathy video?

binarist 3y ago

[dead]

Solvency 3y ago

Because by making the model larger you don't need 64bit precision floats you only need 64 discrete bits.

gemeral 3y ago
Do you mind pointing out where they make the model larger? The paper seems to suggest they are maintaining the same model sizes. > Recent research, such as BitNet, is paving the way for a new era of 1-bit Large Language Models (LLMs). In this work, we introduce a 1-bit LLM variant, namely BitNet b1.58, in which every single parameter (or weight) of the LLM is ternary {-1, 0, 1}. It matches the full-precision (i.e., FP16 or BF16) Transformer LLM with the same model size and training tokens in terms of both perplexity and end-task performance, while being significantly more cost-effective in terms of latency, memory, throughput, and energy consumption

barbarr 3y ago

The activations are still 8-bit, so a lot of complexity and nonlinearity is still expressible. Only the weights are 1.58-bit.

HanClinto 3y ago

On its own, each trit doesn't encode much information at all. But it's not about information at the individual level -- it's more about the shape of the network. I appreciated this comment [0] from earlier in the thread by paul_mk1: > My best guess is that it is encouraging the network to choose good underlying subnetworks to solve the problem, similar to Lottery Ticket Hypothesis. With ternary weights it is just about who connects to who (ie a graph), and not about the individual weight values anymore. For myself, I've done a lot of work with image hashing (such as pHash and dHash) -- and in those, you throw away a LOT of information, but simply by keeping the value of each region and tracking whether or not it's above or below the average (essentially, the sign), then it's astounding how robust those algorithms are. Because you don't look at the individual pixels of an image, but it's very good at capturing the impression of the overall _shape_ of the image. It's less about each individual datum, and more about the shape of the network. If you're not familiar with Lottery Ticket Hypothesis, that would be worth reading up on. [0]: https://news.ycombinator.com/item?id=39544500 https://news.ycombinator.com/item?id=39544500