3 ms·

Neural LMs used to be based on recurrent architectures until the Transformer came along. That architecture is not recursive. I am not sure that a diffusion app

by kleiba 2y ago

Neural LMs used to be based on recurrent architectures until the Transformer came along. That architecture is not recursive. I am not sure that a diffusion approach is all that suitable for generating language. Word are much more discrete than pixels.

WithinReason 2y ago
I meant sequential generation, I didn't mean using an RNN. Diffusion doesn't work on pixels directly either, it works on a latent representation.
- kleiba 2y ago
  All NNs work on latent representations.
  - barrkel 2y ago
    The contrast here is real: there are pixel space diffusion models and latent space diffusion models. Pixel space diffusion is slower because there's more redundant information.