6 ms·
I find this opinion piece rather inane - Investing immense amounts of resources historically helped to find algorithms that could be run with fewer resources.
by sharikous 5y ago
I find this opinion piece rather inane
- Investing immense amounts of resources historically helped to find algorithms that could be run with fewer resources. It's just that the raw computing power helped to lower the cost for research and exploration
- Who is he speaking to? To the big companies pulling out tens of millions to do this? To the small ones? I think everybody knows the costs of deep learning training. The big companies took that into account and still find it cost effective. The small ones already know they cannot afford that power.
- Spectre and Meltdown are bad examples of Moore's law?? I wonder where the author's mind is when he ignores the revolution inherent in stuffing today's tens of billions of transistors in chips that some decade ago contained tens of thousands of them. What has that to do with security vulnerabilities? Granted - those vulnerabilities are in large part due to the increased parallelization needed in today's chips, but that's due to the end of Moore's Law
- As the other commenter said, comparison between neurons and parameters is completely meaningless
- evrydayhustling 5y agoYeah, lots of respect for HuggingFace but this argument is pretty reductive. A very thin slice of the ML community/industry is rushing to short-term commercialization of giant models, and there's nothing wrong with poking some fun at what's being ignored in that haste. But for most of the field, ever-inflating models are a probe for exploring the many types of unreasonable effectiveness of gradient descent. Knowing more about what's possible is incredibly valuable to engineering pragmatic applications of ML. Research and commercialization can happen in parallel, and both sides would be better off if they educated the public about the difference.