3 ms·
It seems like supercomputers are still generally many times more powerful than what AI researchers at top universities or orgs are using. Has any well known AI
by etaioinshrdlu 6y ago
It seems like supercomputers are still generally many times more powerful than what AI researchers at top universities or orgs are using.
Has any well known AI research been done on supercomputers?
It seems like the case that literally just throwing money at the problem is a solid idea nowadays.
- llukas 6y agoCmon. Just look at requirements to train GPT-3 for example... https://lambdalabs.com/blog/demystifying-gpt-3/ https://lambdalabs.com/blog/demystifying-gpt-3/
- etaioinshrdlu 6y agoI guess I had it in my head that they used something like 500 V100's -- which is still a far cry from a big supercomputer these days -- but they haven't published what hardware they used and for how long...
- ydau 6y agoGPT-3 was another buckets worth of evidence in favor of the scaling hypothesis. Performance kept improving (and cost to train kept increasing) as more parameters were added. Even with 175 billion parameters, the performance had not yet plateaued. One take-away is that throwing a lot of compute at the problem helps tremendously :). You can read more about GPT-3 here: https://lambdalabs.com/blog/gpt-3/ https://lambdalabs.com/blog/gpt-3/
- sitkack 6y agoAre you alluding to "The Bitter Lesson" [1] by Rich Sutton [2]? [1] http://incompleteideas.net/IncIdeas/BitterLesson.html http://incompleteideas.net/IncIdeas/BitterLesson.html [2] http://incompleteideas.net/ http://incompleteideas.net/