3 ms·
It looks like a useful article showing the trend OpenAI described a few years ago has continued. One thing I'd object to is coining (or maybe just using?) the
by joe_the_user 5y ago
It looks like a useful article showing the trend OpenAI described a few years ago has continued.
One thing I'd object to is coining (or maybe just using?) the term "Large-Scale Era". This seems like polluting the discussion with a meaningless phrase.
The basic message is essentially, deep learning continues to grow in scale exponentially, many people consider the dividends still worth the costs. No need to "sexy things up" with what no more than a marketing term.
- Tamaybes 5y agoThe result about recent compute trends is different from the recent trends described by OpenAI. In particular, they find a 3.5-month doubling time over the Deep Learning Era, whereas the paper finds a 6-month doubling time. I think the Large-Scale Era does point to a new phenomenon that emerged pretty discontinuously, which is that there are now 'two lanes' in ML scaling. Prior to 2015, academic and industry would train roughly similarly compute intensive models. Since then, a small number of industry players frequently train models with 10-100x more compute than what the typical researcher uses.
- joe_the_user 5y agoThe thing is that the advent of deep learning was a very big change in the sense you had a general purpose method appear that you could use to throw computing power at many/most problems (and tune a bit but still) and get results that previously you couldn't get (and when did get results, you required domain experts). No doubt we have changes within the trajectory of this escalating brute force solutions. But relative changes in this paradigm seem fundamentally different than the initial advent of the paradigm.