3 ms·
How much of it is OpenAI/Microsoft curtailing the compute being used to generate responses?
by RationPhantoms 3y ago
How much of it is OpenAI/Microsoft curtailing the compute being used to generate responses?
- practice9 3y agoThe accuracy loss is more consistent with some kind of quantization of the model(-s) behind the scenes than the alignment gone wrong. Quantization to serve more users faster, on same amount or less of compute.
- arrowsmith 3y agoSorry, what does quantization mean here?
- imdsm 3y agoReducing the precision of the parameters — result being less memory intensive
- iamjackg 3y agoReducing the precision of the weights from high precision floating points to either lower precision floats or even integers. You'd think it would greatly reduce the performance of a model, but in most cases the decline in quality is extremely tolerable compared to the reduction in memory/processing requirements.
- mlboss 3y agoIt means using less number of bits to store float values. This reduces the memory/compute requirement at the cost of making model less precise.