4 ms·
I figured the raw approach of diffusion-esque was tried (without techniques lifted from newer LLMs {chatGPT is only a year old}). I am thinking that those attem
by altruios 3y ago
I figured the raw approach of diffusion-esque was tried (without techniques lifted from newer LLMs {chatGPT is only a year old}). I am thinking that those attempts were made largely before LLM's were popular. Self attention and positional encoding are the claimed 'secret sauce': are there any models (or attempts) anyone can point to that try to combine SA+PE into a diffusion-like (non-autoregressive) model.
What are some models that 'work okay'?
- thatguysaguy 3y agoSelf-attention and positional embeddings are ancient at this point (i.e., more than 5 years old). Here are a couple examples of methods: https://arxiv.org/abs/2112.06749 https://arxiv.org/abs/2112.06749 https://arxiv.org/abs/2205.12558 https://arxiv.org/abs/2205.12558