4 ms·
Optimizing AI Inference at Character.ai
- eachro 2y agoTraining in int8 is noteable (to me). I've been out of date with ML research for a bit now but last I recall, people were mostly training at full precision and then quantizing after training and finetuning a bit on the quantized model afterwards.
- rfoo 2y agoDunno. It could also just mean the so-called "Quantization-aware training" where your weight, activation and gradient is still bf16 and just before use it gets quantized to int8 in the same way you'd do it during inference. This gives you the same "no mismatch between training and predict", and was a standard technique back in vision days (~2018).
- hackernewds 2y agoNoam has been cooking at character.ai in stealth. Their model is impressively engaging
- highlaif 2y ago[dead]
- janalsncm 2y ago> we implemented customized int8 kernels for matrix multiplications and attention I would be curious how this differs from [1] which is supported in Huggingface’s transformers library. [1] https://arxiv.org/abs/2208.07339 https://arxiv.org/abs/2208.07339