2 ms·

Could it be that model just uses latent space for thinking while generating almost garbage? Interesting to check if adding repeating something at the end of pr

• refulgentis 2y ago
  Latent space has a certain shape, which may mean I'm missing a technical distinction.* There's been publications with a pause token (https://arxiv.org/abs/2310.02226 https://arxiv.org/abs/2310.02226), backspace token (https://arxiv.org/abs/2306.05426 https://arxiv.org/abs/2306.05426), or a think token (https://arxiv.org/html/2405.08644v1#Ch0.S4 https://arxiv.org/html/2405.08644v1#Ch0.S4), all of them based on the theory that a generic token can sort of act a placeholder for manipulating attention further without further meaningful output. However, in practice, those approaches haven't been used in the training of a large scale model, i.e. I haven't seen it at all, the most adventurous people have gotten at scale is doing Mamba. (and RL) * It had a particular technical meaning. The first round of the telephone game was when it came to mean "a 3 spatial dimensions-like space, with N dimensions, in an image diffusion model, that contains all possible image styles, that is navigated by a prompt." We're many iterations afield of it now, I'm afraid. Now, you sort of have to interpret it like you would negative space, defined by what it is around it.