7 ms·

We can look Deep Learning's growing demands for computation and despair, or view those growing demands as an economic incentive to develop more powerful hardwar

by cs702 6y ago

We can look Deep Learning's growing demands for computation and despair, or view those growing demands as an economic incentive to develop more powerful hardware that uses energy more efficiently at a lower marginal cost and in a more sustainable manner. In other words, Deep Learning's growing need for computing power seems to have reached a point at which it is now motivating fundamental research to find greener, cheaper, more energy-efficient hardware. The economic incentives are very powerful: Whichever companies (or organizations, or countries) find ways to harness the most computing power at the lowest marginal cost will win the race in this market. -- PS. The same could be said for Bitcoin mining: it is also motivating fundamental research to develop greener, cheaper, more energy-efficient, more powerful hardware. Whoever finds ways to harness the most computing power at the lowest marginal cost will make the most money processing transactions on the network.

rocqua 6y ago
I think the case here is a lot easier than the case for bitcoin mining. Bitcoin miners are so stupidly single purpose that development there doesn't help much. Maybe in general it helps create an industry for designing and manufacturing ASICs. I suppose that might go into making ASICs for deep learning at some point.
- saddlerustle 6y ago
  Bitmain is one of TSMC's largest customers, and that absolutely has been reinvested by TSMC in developing more advanced fabrication techniques. Also bitcoin mining chips are actually a lot like deep learning chips in that it's a lot of simple operations scaled out. And indeed, Bitmain now produces deep learning chips too.
- DaftDank 6y ago
  Can bitcoin ASIC's (or ASIC's in general) be reprogrammed to do another task? I have always heard that they are only designed to do cryptographic hashing, but are they just fundamentally that way? Would programming them to do something else be impossible or very unfeasible?
  - UncleOxidant 6y ago
    ASICs are not reprogrammable. FPGAs are.
    - DaftDank 6y ago
      I'm not well versed in these things so excuse the stupid question, but is the 'code' that runs the chips somehow hard-wired into the chip essentially? I only know vaguely what FPGA's are from investing a little bit in Xilinx late last year, mainly because from what I read they seemed very important to AI implementation and development. What are the benefits of making ASIC's vs FPGA's? Is it a cost thing (i.e. the ASIC's are cheaper to make?)
      - UncleOxidant 6y ago
        The advantage of ASICs is that a design implemented in an ASIC is likely going to run faster than the same design implemented in an FPGA - usually significantly faster. FGPAs are flexible, but as such you trade performance for that flexibility. Developing an ASIC design is very expensive. There's usually many thousands (sometimes hundreds of thousands) of dollars of NRE (Non-Recurring Engineering expenses). Whereas you could put your design into an FPGA for the cost of an FPGA board (typically a few hundred dollars to a few thousand - but you'll be able to use the same FPGA board for many different designs as the FPGA is reprogrammable) and the FPGA design software (which is often given away for free or deeply discounted by the FPGA vendors). So ASICs are for organizations with lots of money. FPGAs are accessible for hobbyists. FPGAs are also often used to test out a design that is eventually implemented in an ASIC. The crypto folks have mostly moved to ASICs for the higher performance.
      - warkdarrior 6y ago
        ASIC are more expensive to make, more performant per watt, and (a lot) less programmable.
      - slimsag 6y ago
        I also won't pretend I know anything here, so take what I say with a grain of salt. From what I understand, FPGAs provide a software method of essentially reprogramming the actual behavior of the physical circuit to perform different tasks. This means FPGAs are (to some extent) general purpose, it's just they are programmed to provide only one purpose at a time (compared to traditional CPUs or GPUs for example). ASICs on the other hand are circuits designed specifically for your application only, they cannot be reprogrammed and must be created specifically for your algorithm.
- glenstein 6y ago
  >Bitcoin miners are so stupidly single purpose that development there doesn't help much. I would say there's a second piece here, too, which is that to the extent that it's driving any sort of research into more efficient practices, it's only in response to constraints of bitcoins own making, and there's every intention to immediately use up new capacity with additional bitcoin mining. What could have been an external net benefit to humanity will just be absorbed into the same things that absorbed prior capacity. I don't want to run all the way with that thought, though, because it can be the case that there's a net benefit to humanity nevertheless. But it makes me think of how Keynes promised that future efficiencies would bring a utopia in the form of dramatically scaling back the need to toil to fulfill basic needs and comforts, how that wasn't necessarily wrong, and how we ended up not taking that path just the same.
peterthehacker 6y ago
Aren’t companies like google[0] and nvidia[1] already doing this? The paper’s point is that eventually we will reach computing power limits and then we will have to improve the deep learning algorithm’s efficiency to continue to improve. From the abstract: > Continued progress in these applications will require dramatically more computationally-efficient methods, which will either have to come from changes to deep learning or from moving to other machine learning methods. [0] https://cloud.google.com/tpu https://cloud.google.com/tpu [1] https://www.nvidia.com/en-us/data-center/v100/ https://www.nvidia.com/en-us/data-center/v100/
- sjg007 6y ago
  Sure. I mean part of it will be finding and exploiting symmetries, improvements to it alternatives to backprop, parallelism as well.
  - hansvm 6y ago
    > finding and exploiting symmetries I'm having trouble finding the papers I've skimmed on this (my original search phrase was something about succinct neural network encodings, entropy, yada yada yada), but that's already being done here and there, at least at a high level looking at graph symmetries (results were okayish -- theoretical space bounds aren't substantial improvements, and the compressed representations didn't perform super well). I haven't seen anything interesting yet explicitly dealing with the fact that neural networks represent the real world in some fashion and have a lot of biases imposed on the values the weights can take. > improvements to it alternatives to backprop I love seeing these come out. There's the no free lunch theorem and all that jazz, but in practice for real networks this can be a huge win. > parallelism as well Not really, at least if I'm understanding the chain of ancestor comments correctly. The arguments are more about the total cost of a given network than the total time to train it. Those are loosely entangled since with low levels of parallelism we're inclined to operate at higher clock speeds or take other energy-inefficient actions, but generally we would expect a parallel algorithm to be no more energy efficient (with respect to total conceptual work performed -- e.g., training a fixed neural network) than an equivalent serial implementation.
jacquesm 6y ago
What is the relative proportion of the number of GPUs now sold for deep learning purposes vs gaming? I think it is definitely true that historically the gamers drove the adoption of GPUs and almost accidentally enabled the deep learning revolution, but has the balance shifted already and is deep learning now the driving force? And are GPUs going to be completely superseded by custom deep learning hardware or rather not?
- phreeza 6y ago
  Nvidia revenue is still slightly more gaming than data center (which covers AI), but the are about to reach parity https://nvidianews.nvidia.com/news/nvidia-announces-financial-results-for-first-quarter-fiscal-2021 https://nvidianews.nvidia.com/news/nvidia-announces-financia...
KKKKkkkk1 6y ago
Saying that increased demand for compute will spur exponential growth is like saying that increased cross-Atlantic travel will spur the development of supersonic passenger airplanes, which we know is the opposite of what happened. Alex Krizhevsky's revolution was to find a way to train large neural networks using existing hardware that was optimized for linear algebra. Linear algebra is literally the first applied problem that was studied in computer science, with papers that were written by Turing and von Neumann. It's the most mature field in computing and is long out of steam. Progress since AlexNet came from scaling $$$ not scaling tflops. There will not be exponential growth in compute performance.
- MacsHeadroom 6y ago
  TPUs and waferscale hardware beg to differ.
  - KKKKkkkk1 6y ago
    Wafer scale has existed since the 1980s. It's literally Concorde-era technology.
  - deepnotderp 6y ago
    1. TPUs haven't really beaten newer GPUs significantly in performance. 2. Wafer scale integration and other advanced packaging approaches are qualitatively different types of computing advances that CMOS scaling, which provides better, faster, smaller, cheaper and at the same time, due to Dennard Scaling, reduces power.
- baxtr 6y ago
  I really don't understand that comment. We had the Concorde, didn't we? We found out that the marginal costs are not worth the 2h saved, fair enough, but there was a race towards faster. And I believe there are companies developing supersonic planes today again, maybe at lower marginal costs. OP did not say the efforts will be successful, OP said there will be a race to see if it is possible to drive costs down spurred by increased demand.
uoaei 6y ago
Deep learning doesn't need to consume more energy. We are doing enough of that as it is. Instead, we need to start moving toward more efficient and specialized implementations of statistical models. SciML is a great example of how we can move past the age of amorphous black boxes and into more efficient representations of the same problems.
raxxorrax 6y ago
Before that is was computer gaming to a large degree. Not yet sure if that isn't still the more reasonable and constructive use of computing power to be honest, although some results of machine learning begin to seem slightly impressive. But I still think that system design is much more important than raw computing power, since the latter is still quite cheaply available on the market.
HourglassFR 6y ago
> In other words, Deep Learning's growing need for computing power seems to have reached a point at which it is now motivating fundamental research to find greener, cheaper, more energy-efficient hardware I don't doubt we will find more frugal hardware energy wise, but calling it greener is somewhat dishonest. > The economic incentives are very powerful: […] Precisely, a technology is not green in itself, it depends on its use within society, and as you pointed out, we are talking about a race here. So the solution is not merely technological, there is some policy and/or societal change to make this greener. Otherwise we are just going to red-queen ourselves into Jevons paradox.
ghj 6y ago
If you think about it, it's insane how much infrastructure we take for granted (xxx kilowatt of electricity, xxx mbps of internet, all for dirt cheap). Maybe in a near future everyone would need xxx petaflops of compute just as you would any other utility. At least anyone who wants their personal AI secretaries to "learn" and "think" effectively. Historically these utility/infrastructure projects needs to be built by the government since it's a chicken and egg problem. "Internet" companies can only exist after the internet is built. "AI" companies can only exist after gpt3 levels of compute is made available to the everyday person. (There might be a physics argument to why the economy of scale of compute can't go down that much even with nation-level of investments. Usually it ends up with needing to build a dyson swarm. I really hope they are wrong because I don't think I will live to see it)
deleted 6y ago
[deleted]
doonesbury 6y ago
BC is a counter example & does not help here at all.

by cs702 6y ago

rocqua 6y ago

I think the case here is a lot easier than the case for bitcoin mining. Bitcoin miners are so stupidly single purpose that development there doesn't help much. Maybe in general it helps create an industry for designing and manufacturing ASICs. I suppose that might go into making ASICs for deep learning at some point.

saddlerustle 6y ago
Bitmain is one of TSMC's largest customers, and that absolutely has been reinvested by TSMC in developing more advanced fabrication techniques. Also bitcoin mining chips are actually a lot like deep learning chips in that it's a lot of simple operations scaled out. And indeed, Bitmain now produces deep learning chips too.
DaftDank 6y ago
Can bitcoin ASIC's (or ASIC's in general) be reprogrammed to do another task? I have always heard that they are only designed to do cryptographic hashing, but are they just fundamentally that way? Would programming them to do something else be impossible or very unfeasible?
- UncleOxidant 6y ago
  ASICs are not reprogrammable. FPGAs are.
  - DaftDank 6y ago
    I'm not well versed in these things so excuse the stupid question, but is the 'code' that runs the chips somehow hard-wired into the chip essentially? I only know vaguely what FPGA's are from investing a little bit in Xilinx late last year, mainly because from what I read they seemed very important to AI implementation and development. What are the benefits of making ASIC's vs FPGA's? Is it a cost thing (i.e. the ASIC's are cheaper to make?)
    - UncleOxidant 6y ago
      The advantage of ASICs is that a design implemented in an ASIC is likely going to run faster than the same design implemented in an FPGA - usually significantly faster. FGPAs are flexible, but as such you trade performance for that flexibility. Developing an ASIC design is very expensive. There's usually many thousands (sometimes hundreds of thousands) of dollars of NRE (Non-Recurring Engineering expenses). Whereas you could put your design into an FPGA for the cost of an FPGA board (typically a few hundred dollars to a few thousand - but you'll be able to use the same FPGA board for many different designs as the FPGA is reprogrammable) and the FPGA design software (which is often given away for free or deeply discounted by the FPGA vendors). So ASICs are for organizations with lots of money. FPGAs are accessible for hobbyists. FPGAs are also often used to test out a design that is eventually implemented in an ASIC. The crypto folks have mostly moved to ASICs for the higher performance.
    - warkdarrior 6y ago
      ASIC are more expensive to make, more performant per watt, and (a lot) less programmable.
    - slimsag 6y ago
      I also won't pretend I know anything here, so take what I say with a grain of salt. From what I understand, FPGAs provide a software method of essentially reprogramming the actual behavior of the physical circuit to perform different tasks. This means FPGAs are (to some extent) general purpose, it's just they are programmed to provide only one purpose at a time (compared to traditional CPUs or GPUs for example). ASICs on the other hand are circuits designed specifically for your application only, they cannot be reprogrammed and must be created specifically for your algorithm.
glenstein 6y ago
>Bitcoin miners are so stupidly single purpose that development there doesn't help much. I would say there's a second piece here, too, which is that to the extent that it's driving any sort of research into more efficient practices, it's only in response to constraints of bitcoins own making, and there's every intention to immediately use up new capacity with additional bitcoin mining. What could have been an external net benefit to humanity will just be absorbed into the same things that absorbed prior capacity. I don't want to run all the way with that thought, though, because it can be the case that there's a net benefit to humanity nevertheless. But it makes me think of how Keynes promised that future efficiencies would bring a utopia in the form of dramatically scaling back the need to toil to fulfill basic needs and comforts, how that wasn't necessarily wrong, and how we ended up not taking that path just the same.

peterthehacker 6y ago

Aren’t companies like google[0] and nvidia[1] already doing this? The paper’s point is that eventually we will reach computing power limits and then we will have to improve the deep learning algorithm’s efficiency to continue to improve. From the abstract: > Continued progress in these applications will require dramatically more computationally-efficient methods, which will either have to come from changes to deep learning or from moving to other machine learning methods. [0] https://cloud.google.com/tpu https://cloud.google.com/tpu [1] https://www.nvidia.com/en-us/data-center/v100/ https://www.nvidia.com/en-us/data-center/v100/

sjg007 6y ago
Sure. I mean part of it will be finding and exploiting symmetries, improvements to it alternatives to backprop, parallelism as well.
- hansvm 6y ago
  > finding and exploiting symmetries I'm having trouble finding the papers I've skimmed on this (my original search phrase was something about succinct neural network encodings, entropy, yada yada yada), but that's already being done here and there, at least at a high level looking at graph symmetries (results were okayish -- theoretical space bounds aren't substantial improvements, and the compressed representations didn't perform super well). I haven't seen anything interesting yet explicitly dealing with the fact that neural networks represent the real world in some fashion and have a lot of biases imposed on the values the weights can take. > improvements to it alternatives to backprop I love seeing these come out. There's the no free lunch theorem and all that jazz, but in practice for real networks this can be a huge win. > parallelism as well Not really, at least if I'm understanding the chain of ancestor comments correctly. The arguments are more about the total cost of a given network than the total time to train it. Those are loosely entangled since with low levels of parallelism we're inclined to operate at higher clock speeds or take other energy-inefficient actions, but generally we would expect a parallel algorithm to be no more energy efficient (with respect to total conceptual work performed -- e.g., training a fixed neural network) than an equivalent serial implementation.

jacquesm 6y ago

What is the relative proportion of the number of GPUs now sold for deep learning purposes vs gaming? I think it is definitely true that historically the gamers drove the adoption of GPUs and almost accidentally enabled the deep learning revolution, but has the balance shifted already and is deep learning now the driving force? And are GPUs going to be completely superseded by custom deep learning hardware or rather not?

phreeza 6y ago
Nvidia revenue is still slightly more gaming than data center (which covers AI), but the are about to reach parity https://nvidianews.nvidia.com/news/nvidia-announces-financial-results-for-first-quarter-fiscal-2021 https://nvidianews.nvidia.com/news/nvidia-announces-financia...

KKKKkkkk1 6y ago

Saying that increased demand for compute will spur exponential growth is like saying that increased cross-Atlantic travel will spur the development of supersonic passenger airplanes, which we know is the opposite of what happened. Alex Krizhevsky's revolution was to find a way to train large neural networks using existing hardware that was optimized for linear algebra. Linear algebra is literally the first applied problem that was studied in computer science, with papers that were written by Turing and von Neumann. It's the most mature field in computing and is long out of steam. Progress since AlexNet came from scaling $$$ not scaling tflops. There will not be exponential growth in compute performance.

MacsHeadroom 6y ago
TPUs and waferscale hardware beg to differ.
- KKKKkkkk1 6y ago
  Wafer scale has existed since the 1980s. It's literally Concorde-era technology.
- deepnotderp 6y ago
  1. TPUs haven't really beaten newer GPUs significantly in performance. 2. Wafer scale integration and other advanced packaging approaches are qualitatively different types of computing advances that CMOS scaling, which provides better, faster, smaller, cheaper and at the same time, due to Dennard Scaling, reduces power.
baxtr 6y ago
I really don't understand that comment. We had the Concorde, didn't we? We found out that the marginal costs are not worth the 2h saved, fair enough, but there was a race towards faster. And I believe there are companies developing supersonic planes today again, maybe at lower marginal costs. OP did not say the efforts will be successful, OP said there will be a race to see if it is possible to drive costs down spurred by increased demand.

uoaei 6y ago

Deep learning doesn't need to consume more energy. We are doing enough of that as it is. Instead, we need to start moving toward more efficient and specialized implementations of statistical models. SciML is a great example of how we can move past the age of amorphous black boxes and into more efficient representations of the same problems.

raxxorrax 6y ago

Before that is was computer gaming to a large degree. Not yet sure if that isn't still the more reasonable and constructive use of computing power to be honest, although some results of machine learning begin to seem slightly impressive. But I still think that system design is much more important than raw computing power, since the latter is still quite cheaply available on the market.

HourglassFR 6y ago

> In other words, Deep Learning's growing need for computing power seems to have reached a point at which it is now motivating fundamental research to find greener, cheaper, more energy-efficient hardware I don't doubt we will find more frugal hardware energy wise, but calling it greener is somewhat dishonest. > The economic incentives are very powerful: […] Precisely, a technology is not green in itself, it depends on its use within society, and as you pointed out, we are talking about a race here. So the solution is not merely technological, there is some policy and/or societal change to make this greener. Otherwise we are just going to red-queen ourselves into Jevons paradox.

ghj 6y ago

If you think about it, it's insane how much infrastructure we take for granted (xxx kilowatt of electricity, xxx mbps of internet, all for dirt cheap). Maybe in a near future everyone would need xxx petaflops of compute just as you would any other utility. At least anyone who wants their personal AI secretaries to "learn" and "think" effectively. Historically these utility/infrastructure projects needs to be built by the government since it's a chicken and egg problem. "Internet" companies can only exist after the internet is built. "AI" companies can only exist after gpt3 levels of compute is made available to the everyday person. (There might be a physics argument to why the economy of scale of compute can't go down that much even with nation-level of investments. Usually it ends up with needing to build a dyson swarm. I really hope they are wrong because I don't think I will live to see it)

deleted 6y ago

[deleted]

doonesbury 6y ago

BC is a counter example & does not help here at all.