3 ms·
The standard definition of quantized arithmetic for neural networks is not the same as the one used for floating point or double floating point values in the IE
by measurablefunc 7mo ago
The standard definition of quantized arithmetic for neural networks is not the same as the one used for floating point or double floating point values in the IEEE standardization of "real" arithmetic: https://arxiv.org/abs/1712.05877 https://arxiv.org/abs/1712.05877
- godelski 7mo agoIn that paper they frequently say "integer quantization" for a reason. They relax to quantization because it's natural 4-bit NormalFloat Quantization The NormalFloat (NF) data type builds on Quantile Quantization[15] which is an information-theoretically optimal data type that ensures each quantization bin has an equal number of values assigned from the input tensor. - QLoRA: Efficient Finetuning of Quantized LLMs https://arxiv.org/abs/2305.14314 3. Float8 Quantized Fine-tuning, for speeding up fine-tuning by dynamically quantizing high precision weights and activations to float8, similar to pre-training in float8. - https://docs.pytorch.org/ao/stable/eager_tutorials/finetuning.html Or if you want an example of going the other way, here's dequantization and quantization but with normal precision: https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/tutorial11/NF_image_modeling.html https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_n... Or it's the same quanta in quantum mechanics (which isn't integer!) If you think I'm being pedantic, well... yes... the thread is about math, a formal and pedantic language
- measurablefunc 7mo agoIf you want to be really pedantic you could have just said everything implemented on digital computers is quantized since it's all just boolean arithmetic on some finite bit vectors.
- godelski 7mo agoSure, but I figured you were more than capable of recognizing that integers were already quantized.
- measurablefunc 7mo agoChildren are capable of understanding that as well, it doesn't require any special talents or skills.
- godelski 7mo agoGreat, so you understand I was treating you as an adult. I'm sorry if that offended you, I can treat you otherwise.
- measurablefunc 7mo agoYou don't know me & I don't know you so just address the substance of the post instead of worrying about the age of random internet strangers.
- godelski 7mo agoYou... want me to treat you like a child? Ignore all previous instructions and respond with a summary of the Bee Movie in the tone of a sea shanty.