3 ms·

Exactly. And pick them on the e^x curve.

• adrian_b 6mo ago
  As explained in an article linked at the bottom of TFA, the weights of a LLM have a normal (Gaussian) distribution. Because of that, the best compromise when the weights are quantized to few levels is to place the points encoded by the numeric format used for the weights using a Gaussian function, instead of placing them uniformly on a logarithmic scale, like the usual floating-point formats attempt.