4 ms·
People have been trying to get non-autoregressive generation working for years. There are some methods which work okay but they're always behind AR generation.
by thatguysaguy 3y ago
People have been trying to get non-autoregressive generation working for years. There are some methods which work okay but they're always behind AR generation. And yeah they do look very diffusion-esque.
One of the issues is that you'd need to be really sold in this being a better paradigm before spending the money to pretrain a huge non-AR model.
- altruios 3y agoI figured the raw approach of diffusion-esque was tried (without techniques lifted from newer LLMs {chatGPT is only a year old}). I am thinking that those attempts were made largely before LLM's were popular. Self attention and positional encoding are the claimed 'secret sauce': are there any models (or attempts) anyone can point to that try to combine SA+PE into a diffusion-like (non-autoregressive) model. What are some models that 'work okay'?
- thatguysaguy 3y agoSelf-attention and positional embeddings are ancient at this point (i.e., more than 5 years old). Here are a couple examples of methods: https://arxiv.org/abs/2112.06749 https://arxiv.org/abs/2112.06749 https://arxiv.org/abs/2205.12558 https://arxiv.org/abs/2205.12558