5 ms·

512 GPUs on 56 Gbps network? I'd rather see researchers exploring potentially more efficient alternatives to traditional neural nets, like XOR Nets, or differen

512 GPUs on 56 Gbps network? I'd rather see researchers exploring potentially more efficient alternatives to traditional neural nets, like XOR Nets, or different architectures like ngForests or probabilistic and logistic circuits, or maybe listen to Vapnik and invest into general statistical learning efficiency.

WrtCdEvrydy 8y ago
Remember this is the regular flow of hardware. First we built awesome high power single core chips, then multi core chips and continued improving performance per dollar. A single GPU in 10 years might be able to smoke these 512 GPUs.
- gambler 8y ago
  >A single GPU in 10 years might be able to smoke these 512 GPUs. Firstly, it will not, because there are physical limitations to silicon-based computing and I don't think we will get a different kind in the next 10-20 years. Secondly, unlike commodity programming, AI is an area where improving algorithm/learning efficiency is infinitely more valuable than figuring out how to throw more hardware at the problem. For starters, any algorithmic improvement makes all further research and experimentation easier for everybody as opposed to a handful of agents who have access to ridiculously powerful hardware. I can list many other reasons, like global energy consumption and the need for learning in embedded devices. I find it highly annoying that deep learning enthusiasts immediately turn into uber-skeptics whenever the conversation touches on other ML approaches. Makes me wonder how difficult it is to get funding for fundamentally new AI research in this climate.
  - WrtCdEvrydy 8y ago
    For reference, 10 years ago, graphics cards were on a 65nm platform while today, we use 14nm graphics cards. The physical limitations may be reached, but we'll just throw more cores unto the PCB to compensate.
    - JustFinishedBSG 8y ago
      We can't even do that, the biggest GPUs today are at the reticle limits of foundries> They literally can't be made bigger. Of course we can still go the chiplet way but...
      - CyberDildonics 8y ago
        But what?
      - frankchn 8y ago
        ... but power requirements would put a limit to that as well. The RTX 2080 Ti already has a 250W TDP. Putting a couple of those on a single card and you are looking at >1000 W. Cooling and power becomes very hard as we are effectively trying to run and cool a space heater at the same time.
  - MasterScrat 8y ago
    > AI is an area where improving algorithm/learning efficiency is infinitely more valuable than figuring out how to throw more hardware at the problem Improving algorithm/learning efficiency gets much easier when you can iterate faster.
ousta 8y ago
welcome to machine learning 101.
ribalda 8y ago
I have realised that the age of a AI is just a movement of big corporations towards a profitable monopoly. They are teaching us how to solve problems with hardware instead with algorithms, and we, as individuals, will never have access to their computing power.
- nl 8y ago
  This is completely wrong. I work in the field, and the way it works is like this, in this order: 1) Someone works out how to do something 2) Someone works out how to improve the accuracy 3) The accuracy maxes out 4) People improve training efficiency. We see this over and over again. Take a look at the FastAI results on ImageNet training speed for example.
- fromthestart 8y ago
  You can easily build and train commercial grade neural nets on consumer hardware. Read about, say, a state of the art image recognition net on arxiv, pull an implementation for python in tensor flow or cafe or pytorch or what have you from GitHub (tons of open source), and with nothing more than just a cursory understanding of what you're doing and some basic programming skills, you can run scripts to train and evaluate functional neural networks on your own data set. I've fully trained numerous modern architectures on a single 1080TI to perform image recognition in a matter of days. All of this is within reach of the average developer. If there's any monopoly, it's over training data, which Google and Amazon happen to specialize in. But even large datasets exist as open source. From what I can tell, machine learning, the precursor to AI, is here, and both knowlege and implementation are fully accessable to the general populace.

WrtCdEvrydy 8y ago

Remember this is the regular flow of hardware. First we built awesome high power single core chips, then multi core chips and continued improving performance per dollar. A single GPU in 10 years might be able to smoke these 512 GPUs.

gambler 8y ago
>A single GPU in 10 years might be able to smoke these 512 GPUs. Firstly, it will not, because there are physical limitations to silicon-based computing and I don't think we will get a different kind in the next 10-20 years. Secondly, unlike commodity programming, AI is an area where improving algorithm/learning efficiency is infinitely more valuable than figuring out how to throw more hardware at the problem. For starters, any algorithmic improvement makes all further research and experimentation easier for everybody as opposed to a handful of agents who have access to ridiculously powerful hardware. I can list many other reasons, like global energy consumption and the need for learning in embedded devices. I find it highly annoying that deep learning enthusiasts immediately turn into uber-skeptics whenever the conversation touches on other ML approaches. Makes me wonder how difficult it is to get funding for fundamentally new AI research in this climate.
- WrtCdEvrydy 8y ago
  For reference, 10 years ago, graphics cards were on a 65nm platform while today, we use 14nm graphics cards. The physical limitations may be reached, but we'll just throw more cores unto the PCB to compensate.
  - JustFinishedBSG 8y ago
    We can't even do that, the biggest GPUs today are at the reticle limits of foundries> They literally can't be made bigger. Of course we can still go the chiplet way but...
    - CyberDildonics 8y ago
      But what?
    - frankchn 8y ago
      ... but power requirements would put a limit to that as well. The RTX 2080 Ti already has a 250W TDP. Putting a couple of those on a single card and you are looking at >1000 W. Cooling and power becomes very hard as we are effectively trying to run and cool a space heater at the same time.
- MasterScrat 8y ago
  > AI is an area where improving algorithm/learning efficiency is infinitely more valuable than figuring out how to throw more hardware at the problem Improving algorithm/learning efficiency gets much easier when you can iterate faster.

ousta 8y ago

welcome to machine learning 101.

ribalda 8y ago

I have realised that the age of a AI is just a movement of big corporations towards a profitable monopoly. They are teaching us how to solve problems with hardware instead with algorithms, and we, as individuals, will never have access to their computing power.

nl 8y ago
This is completely wrong. I work in the field, and the way it works is like this, in this order: 1) Someone works out how to do something 2) Someone works out how to improve the accuracy 3) The accuracy maxes out 4) People improve training efficiency. We see this over and over again. Take a look at the FastAI results on ImageNet training speed for example.
fromthestart 8y ago
You can easily build and train commercial grade neural nets on consumer hardware. Read about, say, a state of the art image recognition net on arxiv, pull an implementation for python in tensor flow or cafe or pytorch or what have you from GitHub (tons of open source), and with nothing more than just a cursory understanding of what you're doing and some basic programming skills, you can run scripts to train and evaluate functional neural networks on your own data set. I've fully trained numerous modern architectures on a single 1080TI to perform image recognition in a matter of days. All of this is within reach of the average developer. If there's any monopoly, it's over training data, which Google and Amazon happen to specialize in. But even large datasets exist as open source. From what I can tell, machine learning, the precursor to AI, is here, and both knowlege and implementation are fully accessable to the general populace.