4 ms·

LLM's generate 'the next token' require a serialization of data, and prevents 'too much' parallelization. I wonder if some sort of diffusion hybrid is being wo

LLM's generate 'the next token' require a serialization of data, and prevents 'too much' parallelization. I wonder if some sort of diffusion hybrid is being worked on. Something that approximates a complete text answer, but 'increases the resolution' of that text over time. One key benefit to a 'shot-gun/top-down/all-at-once' approach as opposed to a 'guess the next token => add and repeat' is the ability for the tokens at the end of the text (a twist in a surprise mystery story) have a direct effect on the beginning of that story. which (as far as I know) is not possible in current LLM's due to their architecture. how self attention and positional encoding would work in a diffusion model... that's the question I have if that could even work... to be clear, I don't mean stable diffusion rendering text as an image. I mean diffusion run on raw (random) text: turning that into a legible response.

thatguysaguy 3y ago
People have been trying to get non-autoregressive generation working for years. There are some methods which work okay but they're always behind AR generation. And yeah they do look very diffusion-esque. One of the issues is that you'd need to be really sold in this being a better paradigm before spending the money to pretrain a huge non-AR model.
- altruios 3y ago
  I figured the raw approach of diffusion-esque was tried (without techniques lifted from newer LLMs {chatGPT is only a year old}). I am thinking that those attempts were made largely before LLM's were popular. Self attention and positional encoding are the claimed 'secret sauce': are there any models (or attempts) anyone can point to that try to combine SA+PE into a diffusion-like (non-autoregressive) model. What are some models that 'work okay'?
  - thatguysaguy 3y ago
    Self-attention and positional embeddings are ancient at this point (i.e., more than 5 years old). Here are a couple examples of methods: https://arxiv.org/abs/2112.06749 https://arxiv.org/abs/2112.06749 https://arxiv.org/abs/2205.12558 https://arxiv.org/abs/2205.12558

thatguysaguy 3y ago

People have been trying to get non-autoregressive generation working for years. There are some methods which work okay but they're always behind AR generation. And yeah they do look very diffusion-esque. One of the issues is that you'd need to be really sold in this being a better paradigm before spending the money to pretrain a huge non-AR model.

altruios 3y ago
I figured the raw approach of diffusion-esque was tried (without techniques lifted from newer LLMs {chatGPT is only a year old}). I am thinking that those attempts were made largely before LLM's were popular. Self attention and positional encoding are the claimed 'secret sauce': are there any models (or attempts) anyone can point to that try to combine SA+PE into a diffusion-like (non-autoregressive) model. What are some models that 'work okay'?
- thatguysaguy 3y ago
  Self-attention and positional embeddings are ancient at this point (i.e., more than 5 years old). Here are a couple examples of methods: https://arxiv.org/abs/2112.06749 https://arxiv.org/abs/2112.06749 https://arxiv.org/abs/2205.12558 https://arxiv.org/abs/2205.12558