4 ms·
As explained in an article linked at the bottom of TFA, the weights of a LLM have a normal (Gaussian) distribution. Because of that, the best compromise when t
by adrian_b 7mo ago
As explained in an article linked at the bottom of TFA, the weights of a LLM have a normal (Gaussian) distribution.
Because of that, the best compromise when the weights are quantized to few levels is to place the points encoded by the numeric format used for the weights using a Gaussian function, instead of placing them uniformly on a logarithmic scale, like the usual floating-point formats attempt.