3 ms·
Treating it as continuous is a property of the training algorithm, but there are networks that use binary values. https://ieeexplore.ieee.org/document/9359148
by astrange 2y ago
Treating it as continuous is a property of the training algorithm, but there are networks that use binary values.
https://ieeexplore.ieee.org/document/9359148 https://ieeexplore.ieee.org/document/9359148
https://arxiv.org/abs/2205.13016 https://arxiv.org/abs/2205.13016
- HarHarVeryFunny 2y agoThose aren't methods of training networks - they are ways to compress (via quantization) networks that have already been trained.
- astrange 2y agoI know. The important thing is how the inference works on them.
- HarHarVeryFunny 2y agoBut we're discussing a training technique, that explicitly takes advantage of the continuous (and embedding vs token probability) representations ... You could quantize a model like this after training, as usual, but that's irrelevant.
- astrange 2y agoThe paper title is "Training Large Language Models to Reason in a Continuous Latent Space". It's true it says training in the title, but the goal (reasoning in continuous space) happens at inference time.