4 ms·
Your second link has the immediate comment "Gpt3 includes dense attention layers that are n^2". So it's not at all unlikely.
by MichaelZuo 4y ago
Your second link has the immediate comment "Gpt3 includes dense attention layers that are n^2". So it's not at all unlikely.
- space_fountain 4y agoGPT3 was released 3 years ago now. There have been major advancements in scaling attention so it would be strange if they didn't use some of them
- MichaelZuo 4y agoIt doesn't matter how many major advancements they made in scaling, as long as one component is O(n^2) or higher.
- cubefox 4y agoIt's not the scale itself, it's the scaling architecture.
- MichaelZuo 4y agoThe same applies.