3 ms·
Similarly, ChatGPT (gpt2-medium) has 12 heads of attention. GPT-3 has 96.
by politician 4y ago
Similarly, ChatGPT (gpt2-medium) has 12 heads of attention. GPT-3 has 96.
- IIAOPSW 4y agoI don't know how superimposed waves in finely tuned timing loops with non-linear interference translates into heads of attention, and honestly I suspect a lot of the things difficult to do with heads of attention (and other approaches of the past) come for free in a resonance based system.