3 ms·

1-bit is not much though. Here's what I got: Me: > Describe the process of pasteurisation Response: > pasteurization is a, which a, the which which is, the

1-bit is not much though. Here's what I got: Me: > Describe the process of pasteurisation Response: > pasteurization is a, which a, the which which is, the and process, the the past, the and the, and and and past, and past, and and and and the, the and past, process is the is is is is, and the, past, and and future, process, and and and and and and and the, past, process, process, the the, the the, is and the, the the, and and and, and process, and and the, past, and and and past, and the, the, and the past, the the, process, process, and past, the past, past, the and, the past, and and and and and and and and and the, and and and and and and and, the the, the the, the or and and, the the, the and the, which the, and the, the the, past, and n the process, and and and and, past, and, and and, the past, and, the the, past,, and the, the the, the is the, past, and and and and and, and and and past, the and the, the the, the the, are and past, and which the, the and n, n the, the n past, past, n the, and n, the the process, which past, the the, the n, the the, the is past, the the, is, past, the the, past, and past, process, the the, the the, the and and the, and which past, the (and that basically just goes on and on like that)

wongarsu 3mo ago
I got a decent response with the same prompt But yes, this will happen. 1.7B is already a tiny model, running that at 1-bit is going to lead to some behavioral issues and will need tuning of temperature/topP/repeatPenalty/etc, and probably a software fallback that detects pathological cases like this and simply retries. Just like we all had to do in the times of GPT 2 and GPT 3
- OutOfHere 3mo ago
  I confirm I got an extremely detailed and long response for "Describe the process of pasteurization." with 1.7B. I had to tune nothing.
nikhizzle 3mo ago
IMHO from guy working with heavily quantized models and trying to get results. I do think quantization at less than 4-bit requires more advanced mathematics than is usually used. See Google Scholar Babak Hassibi (founder of Prism ML). Generally am amazed about the resiliency of these new data structures to approximation.
- cubefox 3mo ago
  There was also a NeurIPS paper which successfully trained 1-bit models directly without floating point weights or any quantization: https://arxiv.org/abs/2405.16339 https://arxiv.org/abs/2405.16339 I didn't see any further work in this direction however. The paper seems to have been mostly overlooked.
  - adrian_b 3mo ago
    While what can be done with less bits per parameter may be interesting, this is not really the important number for an LLM. Much more important is the total amount of memory needed by an LLM, which can be decreased either by reducing the number of bits per parameter or the number of parameters. Arithmetic circuits are already cheap so for the inference cost the amount of memory transfers is more important. It is not clear yet which number of bits per parameter allows the minimization of the total memory requirement, at a given LLM quality.
    - cubefox 3mo ago
      I thought the multiplications during training (backpropagation) are quite computationally expensive. The method from the paper above doesn't need any multiplications.

wongarsu 3mo ago

I got a decent response with the same prompt But yes, this will happen. 1.7B is already a tiny model, running that at 1-bit is going to lead to some behavioral issues and will need tuning of temperature/topP/repeatPenalty/etc, and probably a software fallback that detects pathological cases like this and simply retries. Just like we all had to do in the times of GPT 2 and GPT 3

OutOfHere 3mo ago
I confirm I got an extremely detailed and long response for "Describe the process of pasteurization." with 1.7B. I had to tune nothing.

nikhizzle 3mo ago

IMHO from guy working with heavily quantized models and trying to get results. I do think quantization at less than 4-bit requires more advanced mathematics than is usually used. See Google Scholar Babak Hassibi (founder of Prism ML). Generally am amazed about the resiliency of these new data structures to approximation.

cubefox 3mo ago
There was also a NeurIPS paper which successfully trained 1-bit models directly without floating point weights or any quantization: https://arxiv.org/abs/2405.16339 https://arxiv.org/abs/2405.16339 I didn't see any further work in this direction however. The paper seems to have been mostly overlooked.
- adrian_b 3mo ago
  While what can be done with less bits per parameter may be interesting, this is not really the important number for an LLM. Much more important is the total amount of memory needed by an LLM, which can be decreased either by reducing the number of bits per parameter or the number of parameters. Arithmetic circuits are already cheap so for the inference cost the amount of memory transfers is more important. It is not clear yet which number of bits per parameter allows the minimization of the total memory requirement, at a given LLM quality.
  - cubefox 3mo ago
    I thought the multiplications during training (backpropagation) are quite computationally expensive. The method from the paper above doesn't need any multiplications.