3 ms·
O(n^2) seems unlikely: https://cognitiverevolution.substack.com/p/openais-foundry-leaked-pricing-says#:~:text=Aside,price https://cognitiverevolution.substack.
by cubefox 4y ago
O(n^2) seems unlikely:
https://cognitiverevolution.substack.com/p/openais-foundry-leaked-pricing-says#:~:text=Aside,price https://cognitiverevolution.substack.com/p/openais-foundry-l....
https://news.ycombinator.com/item?id=34977194#:~:text=Sparse,%29%29 https://news.ycombinator.com/item?id=34977194#:~:text=Sparse...
- MichaelZuo 4y agoYour second link has the immediate comment "Gpt3 includes dense attention layers that are n^2". So it's not at all unlikely.
- space_fountain 4y agoGPT3 was released 3 years ago now. There have been major advancements in scaling attention so it would be strange if they didn't use some of them
- MichaelZuo 4y agoIt doesn't matter how many major advancements they made in scaling, as long as one component is O(n^2) or higher.
- cubefox 4y agoIt's not the scale itself, it's the scaling architecture.
- MichaelZuo 4y agoThe same applies.