5 ms·

Ride the Hardware Lottery

30 points by EvgeniyZh 6y ago

sdenton4 6y ago
Here's the 'Hardware Lottery' essay that this refers to: https://arxiv.org/pdf/2009.06489.pdf https://arxiv.org/pdf/2009.06489.pdf "Hardware, systems and algorithms research communities have historically had different incentive structures and fluctuating motivation to engage with each other explicitly. This historical treatment is odd given that hardware and software have frequently determined which research ideas succeed (and fail). This essay introduces the term hardware lottery to describe when a research idea wins because it is suited to the available software and hardware and not because the idea is superior to alternative research directions. Examples from early computer science history illustrate how hardware lotteries can delay research progress by casting successful ideas as failures. These lessons are particularly salient given the advent of domain specialized hardware which make it increasingly costly to stray off of the beaten path of research ideas. This essay posits that the gains from progress in computing are likely to become even more uneven, with certain research directions moving into the fast-lane while progress on others is further obstructed."
yunusabd 6y ago
> Despite an electric shaver-like power efficiency, the human brain has limits that some of the Very Large Parameter models transcend (even if that’s unreliable today). Are there any examples where a VLP model actually transcends the limits of the human brain?
- comex 6y ago
  GPT-3 has an encyclopedic knowledge base that no (single) human brain could match. It loses to human brains in most aspects, but it wins in that one. Other than that… well, I don’t even know of any other VLP models. But there are other examples of ML models in general transcending humans, like Watson in Jeopardy, and AlphaGo.
  - yunusabd 6y ago
    True, but in the context of computer science, isn't having encyclopedic knowledge a trivial problem? I mean, if you look at it that way, wikipedia itself "transcends" the limits of the human brain, and it's just a database. And a way more efficient one at that, at 50GB vs. 700GB. In the context of AI, Bert already had knowledge embedded into it [1], and it's not considered a VLP model afaik. So it's more of a quantitative improvement, contrary to the claim of the author that there's a qualitative difference between VLP models and everything else. [1] https://syncedreview.com/2020/08/12/how-smart-is-bert-evaluating-the-language-models-commonsense-knowledge/ https://syncedreview.com/2020/08/12/how-smart-is-bert-evalua...

30 points by EvgeniyZh 6y ago

Here's the 'Hardware Lottery' essay that this refers to: https://arxiv.org/pdf/2009.06489.pdf https://arxiv.org/pdf/2009.06489.pdf "Hardware, systems and algorithms research communities have historically had different incentive structures and fluctuating motivation to engage with each other explicitly. This historical treatment is odd given that hardware and software have frequently determined which research ideas succeed (and fail). This essay introduces the term hardware lottery to describe when a research idea wins because it is suited to the available software and hardware and not because the idea is superior to alternative research directions. Examples from early computer science history illustrate how hardware lotteries can delay research progress by casting successful ideas as failures. These lessons are particularly salient given the advent of domain specialized hardware which make it increasingly costly to stray off of the beaten path of research ideas. This essay posits that the gains from progress in computing are likely to become even more uneven, with certain research directions moving into the fast-lane while progress on others is further obstructed."

yunusabd 6y ago

> Despite an electric shaver-like power efficiency, the human brain has limits that some of the Very Large Parameter models transcend (even if that’s unreliable today). Are there any examples where a VLP model actually transcends the limits of the human brain?

comex 6y ago
GPT-3 has an encyclopedic knowledge base that no (single) human brain could match. It loses to human brains in most aspects, but it wins in that one. Other than that… well, I don’t even know of any other VLP models. But there are other examples of ML models in general transcending humans, like Watson in Jeopardy, and AlphaGo.
- yunusabd 6y ago
  True, but in the context of computer science, isn't having encyclopedic knowledge a trivial problem? I mean, if you look at it that way, wikipedia itself "transcends" the limits of the human brain, and it's just a database. And a way more efficient one at that, at 50GB vs. 700GB. In the context of AI, Bert already had knowledge embedded into it [1], and it's not considered a VLP model afaik. So it's more of a quantitative improvement, contrary to the claim of the author that there's a qualitative difference between VLP models and everything else. [1] https://syncedreview.com/2020/08/12/how-smart-is-bert-evaluating-the-language-models-commonsense-knowledge/ https://syncedreview.com/2020/08/12/how-smart-is-bert-evalua...