3 ms·

Why is it "continuous" thought? I don't see what is continuous - the values inside an LLM are discrete even if they're floating point. Hmm, I guess you could e

Why is it "continuous" thought? I don't see what is continuous - the values inside an LLM are discrete even if they're floating point. Hmm, I guess you could evaluate it at any given finite precision, but it would be surprising to me if that made it more accurate.

mkl 2y ago
It's far more continuous than constantly jumping to the nearest token vector. The fact that real numbers are approximated by floating point isn't really relevant.
layer8 2y ago
If you are continuously complaining, does it mean you do it non-discretely and with infinite precision?
- astrange 2y ago
  It apparently uses the same iteration strategy as tokenized thinking, so that's not it. > Since both strategies provided comparable results, the researchers opted for using a constant number of thoughts for simplicity.
HarHarVeryFunny 2y ago
> the values inside an LLM are discrete even if they're floating point. If that were true they'd never be able to learn anything - neural nets depend on continuous gradients to learn. Weights get updated by incremental/continuous amounts based on gradients. Even at the output of an LLM, where the internal embeddings have been mapped to token probabilities, those probabilities are also continuous. It's only when you sample from the model that a continuous probability becomes a discrete chosen token.
- astrange 2y ago
  Treating it as continuous is a property of the training algorithm, but there are networks that use binary values. https://ieeexplore.ieee.org/document/9359148 https://ieeexplore.ieee.org/document/9359148 https://arxiv.org/abs/2205.13016 https://arxiv.org/abs/2205.13016
  - HarHarVeryFunny 2y ago
    Those aren't methods of training networks - they are ways to compress (via quantization) networks that have already been trained.
    - astrange 2y ago
      I know. The important thing is how the inference works on them.
      - HarHarVeryFunny 2y ago
        But we're discussing a training technique, that explicitly takes advantage of the continuous (and embedding vs token probability) representations ... You could quantize a model like this after training, as usual, but that's irrelevant.
        astrange 2y ago
        The paper title is "Training Large Language Models to Reason in a Continuous Latent Space". It's true it says training in the title, but the goal (reasoning in continuous space) happens at inference time.

mkl 2y ago

It's far more continuous than constantly jumping to the nearest token vector. The fact that real numbers are approximated by floating point isn't really relevant.

layer8 2y ago

If you are continuously complaining, does it mean you do it non-discretely and with infinite precision?

astrange 2y ago
It apparently uses the same iteration strategy as tokenized thinking, so that's not it. > Since both strategies provided comparable results, the researchers opted for using a constant number of thoughts for simplicity.

HarHarVeryFunny 2y ago

> the values inside an LLM are discrete even if they're floating point. If that were true they'd never be able to learn anything - neural nets depend on continuous gradients to learn. Weights get updated by incremental/continuous amounts based on gradients. Even at the output of an LLM, where the internal embeddings have been mapped to token probabilities, those probabilities are also continuous. It's only when you sample from the model that a continuous probability becomes a discrete chosen token.

astrange 2y ago
Treating it as continuous is a property of the training algorithm, but there are networks that use binary values. https://ieeexplore.ieee.org/document/9359148 https://ieeexplore.ieee.org/document/9359148 https://arxiv.org/abs/2205.13016 https://arxiv.org/abs/2205.13016
- HarHarVeryFunny 2y ago
  Those aren't methods of training networks - they are ways to compress (via quantization) networks that have already been trained.
  - astrange 2y ago
    I know. The important thing is how the inference works on them.
    - HarHarVeryFunny 2y ago
      But we're discussing a training technique, that explicitly takes advantage of the continuous (and embedding vs token probability) representations ... You could quantize a model like this after training, as usual, but that's irrelevant.
      - astrange 2y ago
        The paper title is "Training Large Language Models to Reason in a Continuous Latent Space". It's true it says training in the title, but the goal (reasoning in continuous space) happens at inference time.