3 ms·

Huh? What are you talking about? The escalating compute involved in DL is obvious to anyone reading the papers; OA is just doing the work of putting numbers on

by gwern 8y ago

Huh? What are you talking about? The escalating compute involved in DL is obvious to anyone reading the papers; OA is just doing the work of putting numbers on the trend.

PeterisP 8y ago
There's lots of research on doing the same learning with much less resources (e.g. recent paper https://eng.uber.com/accelerated-neuroevolution/ https://eng.uber.com/accelerated-neuroevolution/ , or the example visible in this very article of AlphaZero having much, much less compute than AlphaGoZero and doing better anyways), and even without that simple hardware progress means that random gaming GPUs can handle datasets that were inconvenient a few years ago. I'd say it all depends on the size of datasets - some domains (e.g. unlabeled image data) have "effectively infinite" datasets where the amount of data you can use is limited only by your computing power, but in many other use cases all the data you'll ever get can be processed by a single beefy workstation. More available compute means that we tackle more difficult problems. However, for any single given task it's often not the case that the amount of compute grows. If anything, the graph is not showing the compute required for DL, but the compute available for DL - it gets used simply because it's there.
- geoffreyirving 8y ago
  Yep, we're not claiming this is the compute required for DL, and for specific tasks we expect compute required to fall over time. But better algorithms actually mean compute is more important, not less, and would likely make the growth in available compute more important. For example, if a task is parameterized (by size or difficulty, say), then a better algorithm might change the asymptotic complexity from O(n^3) to O(n^2). A 2x compute increase for the old algorithm would take us from n -> 1.25n, but the new algorithm would go from n -> 1.41n.
- gwern 8y ago
  > There's lots of research on doing the same learning with much less resources (e.g. recent paper https://eng.uber.com/accelerated-neuroevolution/ https://eng.uber.com/accelerated-neuroevolution/ , or the example visible in this very article of AlphaZero having much, much less compute than AlphaGoZero and doing better anyways), and even without that AlphaZero could not have been created without going through many many iterations of AlphaGo, each one of which cost several GPU-years, and calling AlphaZero cheap is serious moving of goalposts as it required thousands of TPUs for days, and Facebook's recent replication for chess also used thousands of GPUs for 3 weeks. (Zero is cheap only in comparison to the previous AlphaGos using weeks or months of hundreds/thousands of GPUs/TPUs.) Note that that is a log graph; flip over the linear scale and you get an idea of how extraordinarily expensive Zero is compared to everything not named 'AlphaGo'. If anything, this observation implies that AI risk is more dangerous than thought because it implies a 'hardware overhang': it will take vast computational resources to create the first slow inefficient AI but it will then rapidly be optimized (either by itself or human researchers) and able to run far faster/more copies/on more devices/for less money, experiencing a sudden burst in capabilities. Like model compression/distillation where you can take the slow big model you normally train and then turn it into something which is 10x faster or 100x smaller or just plain performs better (see 'born again networks' or ensembling). > simple hardware progress means that random gaming GPUs can handle datasets that were inconvenient a few years ago. ...which means using a lot more compute, yes.
  - geoffreyirving 8y ago
    And it's worth emphasizing that we don't need to have particularly high confidence that this trend continues for it to motivate working on AI safety. All we need is the lack of high confidence that it will stop.
- damodei 8y ago
  It's actually incorrect that AZ got better results that AGZ with less compute. The graph shows the large AGZ, which somewhat exceeded the rating of both the small AGZ and AZ. AZ did slightly outperform the small AGZ, but did so while using a similar amount of compute. On the broader point though, I agree with this. We say that compute and algorithms are complementary in the post. Much of the time, when you come up with an algorithm that allows you to do something that used to cost X compute in 0.2X compute instead, you can use the new algorithm to do something significantly more impressive with the full X compute.
  - adamnemecek 8y ago
    How can I contact you? I have some questions about adversariality in AI.

by gwern 8y ago

Huh? What are you talking about? The escalating compute involved in DL is obvious to anyone reading the papers; OA is just doing the work of putting numbers on the trend.

PeterisP 8y ago

There's lots of research on doing the same learning with much less resources (e.g. recent paper https://eng.uber.com/accelerated-neuroevolution/ https://eng.uber.com/accelerated-neuroevolution/ , or the example visible in this very article of AlphaZero having much, much less compute than AlphaGoZero and doing better anyways), and even without that simple hardware progress means that random gaming GPUs can handle datasets that were inconvenient a few years ago. I'd say it all depends on the size of datasets - some domains (e.g. unlabeled image data) have "effectively infinite" datasets where the amount of data you can use is limited only by your computing power, but in many other use cases all the data you'll ever get can be processed by a single beefy workstation. More available compute means that we tackle more difficult problems. However, for any single given task it's often not the case that the amount of compute grows. If anything, the graph is not showing the compute required for DL, but the compute available for DL - it gets used simply because it's there.

geoffreyirving 8y ago
Yep, we're not claiming this is the compute required for DL, and for specific tasks we expect compute required to fall over time. But better algorithms actually mean compute is more important, not less, and would likely make the growth in available compute more important. For example, if a task is parameterized (by size or difficulty, say), then a better algorithm might change the asymptotic complexity from O(n^3) to O(n^2). A 2x compute increase for the old algorithm would take us from n -> 1.25n, but the new algorithm would go from n -> 1.41n.
gwern 8y ago
> There's lots of research on doing the same learning with much less resources (e.g. recent paper https://eng.uber.com/accelerated-neuroevolution/ https://eng.uber.com/accelerated-neuroevolution/ , or the example visible in this very article of AlphaZero having much, much less compute than AlphaGoZero and doing better anyways), and even without that AlphaZero could not have been created without going through many many iterations of AlphaGo, each one of which cost several GPU-years, and calling AlphaZero cheap is serious moving of goalposts as it required thousands of TPUs for days, and Facebook's recent replication for chess also used thousands of GPUs for 3 weeks. (Zero is cheap only in comparison to the previous AlphaGos using weeks or months of hundreds/thousands of GPUs/TPUs.) Note that that is a log graph; flip over the linear scale and you get an idea of how extraordinarily expensive Zero is compared to everything not named 'AlphaGo'. If anything, this observation implies that AI risk is more dangerous than thought because it implies a 'hardware overhang': it will take vast computational resources to create the first slow inefficient AI but it will then rapidly be optimized (either by itself or human researchers) and able to run far faster/more copies/on more devices/for less money, experiencing a sudden burst in capabilities. Like model compression/distillation where you can take the slow big model you normally train and then turn it into something which is 10x faster or 100x smaller or just plain performs better (see 'born again networks' or ensembling). > simple hardware progress means that random gaming GPUs can handle datasets that were inconvenient a few years ago. ...which means using a lot more compute, yes.
- geoffreyirving 8y ago
  And it's worth emphasizing that we don't need to have particularly high confidence that this trend continues for it to motivate working on AI safety. All we need is the lack of high confidence that it will stop.
damodei 8y ago
It's actually incorrect that AZ got better results that AGZ with less compute. The graph shows the large AGZ, which somewhat exceeded the rating of both the small AGZ and AZ. AZ did slightly outperform the small AGZ, but did so while using a similar amount of compute. On the broader point though, I agree with this. We say that compute and algorithms are complementary in the post. Much of the time, when you come up with an algorithm that allows you to do something that used to cost X compute in 0.2X compute instead, you can use the new algorithm to do something significantly more impressive with the full X compute.
- adamnemecek 8y ago
  How can I contact you? I have some questions about adversariality in AI.