4 ms·
You are conflating post training quantization and low bit training.
by danielmarkbruce 10d ago
You are conflating post training quantization and low bit training.
- kadushka 10d agoThat's what I meant - we are currently use fp4 formats for training, and we cannot quite get away with that, despite dynamic quant and small block size - we still have to use quite a bit of higher precision (fp8 or even fp16) in various model components.
- danielmarkbruce 10d agoI might still be misunderstanding what you are saying, but bitnet also keeps high precision latent weights during training. The optimizer updates those, while the weights used in the forward pass are quantized to ternary values.