3 ms·
The Hardware Lottery (2020)
- winterismute 3y agoI am looking at solving this challenge in a specific way: using high-perf, GPU-accelerated HW simulators and ML algorithms to tune a new HW architecture automatically. Best ML HW => run on it the best ML models => produce new best HW (arch) => build new best HW => GOTO 10. Reach out if you are interested in any way.
- heyitsguay 3y agoHow would you define "new best HW"? Seems challenging, particularly if it's for generation N+1 ML models that haven't been created yet. Also, while there is work underway to use ML models to guide HW design, it's not clear to me that the best ML models for that == the best ML models for more general purpose tasks in audio, visual, and natural language processing, i.e. is HW circuit design done using transformers? What are the inputs, latent space, and outputs? GPU development seems driven by more general computational principles that might be summed up glibly as "We're hitting fundamental physics limits for single-core processors, what is the maximum amount of data we can move per second through the maximum number of cores?" Perhaps there is a way to extend this with a view toward the challenges of current ML model training and inference, but I imagine trying to approach it as a black-box optimization problem could be quite difficult.
- sassy_quat 3y ago[dead]
- PeterisP 3y agoThe issue is that there is no such thing such as "best ML HW" because different ML algorithms need quite different things. The process you describe would work to produce hardware that's best for some specific type of ML models, while the whole point of this article is that the currently "best" ML models are there not because they are objectively best, but rather because they are what's best within the bounds of the hardware architectures we have (i.e. we're not looking at best ML algorithms but rather at ML algorithms that "won the hardware lottery"), and it seems plausible that there might be better ML methods if there was different hardware that enabled them - but the process you describe wouldn't possibly find that different hardware, it would instead tune the hardware even more towards the algorithm path we're already stuck on.
- shae 3y agoI wish for graph reduction hardware, we could build amazing things with that.
- LargoLasskhyfv 3y agohttps://www.cs.york.ac.uk/fp/reduceron/ https://www.cs.york.ac.uk/fp/reduceron/ https://www.cambridge.org/core/journals/journal-of-functional-programming/article/reduceron-reconfigured-and-reevaluated/9818E081664ADAFE9F61F1AEDAD0B043 https://www.cambridge.org/core/journals/journal-of-functiona... https://github.com/tommythorn/Reduceron https://github.com/tommythorn/Reduceron also: https://en.wikipedia.org/wiki/Graph_reduction_machine https://en.wikipedia.org/wiki/Graph_reduction_machine
- jacquesm 3y agoI was about to post that third link as an answer, but you beat me to it by 8 hours :) Anyway, I think an FPGA is an excellent piece of kit to start such development.
- voiceblue 3y agoRelated: The Bitter Lesson [0], and previous discussion [1]. [0] http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html [1] https://news.ycombinator.com/item?id=19393432 https://news.ycombinator.com/item?id=19393432
- dang 3y agoThanks! Macroexpanded: The Bitter Lesson: General Computation Methods Are the Most Effective - https://news.ycombinator.com/item?id=37893468 https://news.ycombinator.com/item?id=37893468 - Oct 2023 (1 comment) The Bitter Lesson (2019) - https://news.ycombinator.com/item?id=37139223 https://news.ycombinator.com/item?id=37139223 - Aug 2023 (1 comment) The Bitter Lesson (2019) [pdf] - https://news.ycombinator.com/item?id=37129921 https://news.ycombinator.com/item?id=37129921 - Aug 2023 (60 comments) The Bitter Lesson – Rich Sutton (2019) - https://news.ycombinator.com/item?id=36017857 https://news.ycombinator.com/item?id=36017857 - May 2023 (8 comments) The Bitter Lesson (2019) - https://news.ycombinator.com/item?id=34006388 https://news.ycombinator.com/item?id=34006388 - Dec 2022 (33 comments) The Bitter Lesson (2019) - https://news.ycombinator.com/item?id=30889873 https://news.ycombinator.com/item?id=30889873 - April 2022 (37 comments) The Bitter Lesson (2019) - https://news.ycombinator.com/item?id=23781400 https://news.ycombinator.com/item?id=23781400 - July 2020 (85 comments) The Bitter Lesson - https://news.ycombinator.com/item?id=19393432 https://news.ycombinator.com/item?id=19393432 - March 2019 (53 comments)
- dang 3y agoRelated: The Hardware Lottery - https://news.ycombinator.com/item?id=29409415 https://news.ycombinator.com/item?id=29409415 - Dec 2021 (1 comment) Ride the Hardware Lottery - https://news.ycombinator.com/item?id=25043592 https://news.ycombinator.com/item?id=25043592 - Nov 2020 (4 comments) The Hardware Lottery - https://news.ycombinator.com/item?id=24515035 https://news.ycombinator.com/item?id=24515035 - Sept 2020 (16 comments)