27 ms·

This is the same argument that mainframe companies made in the 80s. Mass commodity systems will eventually kill the big gpu except for legacy and niche workload

by PLenz 2y ago

This is the same argument that mainframe companies made in the 80s. Mass commodity systems will eventually kill the big gpu except for legacy and niche workloads. It's the tech infra circle of life.

cogman10 2y ago
I'm less optimistic. Unlike the 80s we are quickly approaching the limits of node feature sizes. Barring radically new designs and materials, I think it's reasonable to assume that we are reaching the power:computational limits of silicon.
- latchkey 2y ago
  Maybe in the case of CPUs. But, there is a fundamental technical shift going on away from computationally expensive things being done on CPUs and moving to AIAs. I suspect that this is why Jensen said Moore's law is dead.
  - refulgentis 2y ago
    Jensen said Moore's Law is dead because of the dichotomy explained a few times -- the free lunches are coming fewer and far between, you can't get a substantial performance boost and power decrease every 18 months, per Moore's Law. We are reaching the power:computational limits of silicon. Both GPU and CPU.
    - latchkey 2y ago
      Here is the quote: “Moore’s Law’s dead,” Huang said, referring to the standard that the number of transistors on a chip doubles every two years. “And the ability for Moore’s Law to deliver twice the performance at the same cost, or at the same performance, half the cost, every year and a half, is over. It’s completely over, and so the idea that a chip is going to go down in cost over time, unfortunately, is a story of the past. Computing is a not a chip problem, it’s a software and chip problem,” Huang said. What we are seeing now are software engineers offloading computationally expensive workloads to AIA's, more and more. This is enabled through the use of libraries like PyTorch and access to HPC levels of compute that were not broadly accessible before.
      - MichaelZuo 2y ago
        The 'AIA's also have a scaling limit, and the coordination overhead increases exponentially too. e.g. A million chickens could have a thousand times more muscle mass than two strong oxen, but I doubt anyone could plow a field even 2x faster with a million chickens. So it's only a net benefit when the work can be split up into tiny parallel chunks and then recombined with near perfect efficiently.
        latchkey 2y ago
        > So it's only a net benefit when the work can be split up into tiny parallel chunks and then recombined with near perfect efficiently. Hasn't that pretty much always been the case? One thing we are seeing more and more of is composable fabrics where the PCIe bus is effectively being extended outside the case such that you log into a single instance, and instead of seeing just 8 AIA's, you now see 32+. This makes the coordination a lot easier.
        refulgentis 2y ago
        I don't know how you keep rewording simple things everyone knows and are being patiently explained to you, as if you are encountering them the first time in the thread.
        latchkey 2y ago
        Which composable fabric am I talking about then?
        refulgentis 2y ago
        >> So it's only a net benefit when the work can be split up into tiny parallel chunks and then recombined with near perfect efficiently. > Hasn't that pretty much always been the case? I was talking about that, the fabric stuff is a non-sequitor, random hardware, doesn't make GPUs 2x in speed every 18 months, much less lead to a shift in everyday computing loads to GPUs. 1. Moore's Law also applies to GPUs. 2. If we could make use of 1000 cores for anything but long tail tasks, ye average CPU would have a lot more than 8 cores by now. That's what the million chickens thing is about. You can give me ultra-unobtianium fabric for "AIAs", it doesn't matter unless I have a algorithm that's massively parallelizable.
fragmede 2y ago
What's different today is the divide between training and inference. Inference is ridiculously cheap compared to training, and we're still early days with optimizations across the whole of the stack, so we'll have to see how it develops. Once constant training gets figured out, then we're really in for a ride.

by PLenz 2y ago

This is the same argument that mainframe companies made in the 80s. Mass commodity systems will eventually kill the big gpu except for legacy and niche workloads. It's the tech infra circle of life.

cogman10 2y ago

I'm less optimistic. Unlike the 80s we are quickly approaching the limits of node feature sizes. Barring radically new designs and materials, I think it's reasonable to assume that we are reaching the power:computational limits of silicon.

latchkey 2y ago
Maybe in the case of CPUs. But, there is a fundamental technical shift going on away from computationally expensive things being done on CPUs and moving to AIAs. I suspect that this is why Jensen said Moore's law is dead.
- refulgentis 2y ago
  Jensen said Moore's Law is dead because of the dichotomy explained a few times -- the free lunches are coming fewer and far between, you can't get a substantial performance boost and power decrease every 18 months, per Moore's Law. We are reaching the power:computational limits of silicon. Both GPU and CPU.
  - latchkey 2y ago
    Here is the quote: “Moore’s Law’s dead,” Huang said, referring to the standard that the number of transistors on a chip doubles every two years. “And the ability for Moore’s Law to deliver twice the performance at the same cost, or at the same performance, half the cost, every year and a half, is over. It’s completely over, and so the idea that a chip is going to go down in cost over time, unfortunately, is a story of the past. Computing is a not a chip problem, it’s a software and chip problem,” Huang said. What we are seeing now are software engineers offloading computationally expensive workloads to AIA's, more and more. This is enabled through the use of libraries like PyTorch and access to HPC levels of compute that were not broadly accessible before.
    - MichaelZuo 2y ago
      The 'AIA's also have a scaling limit, and the coordination overhead increases exponentially too. e.g. A million chickens could have a thousand times more muscle mass than two strong oxen, but I doubt anyone could plow a field even 2x faster with a million chickens. So it's only a net benefit when the work can be split up into tiny parallel chunks and then recombined with near perfect efficiently.
      - latchkey 2y ago
        > So it's only a net benefit when the work can be split up into tiny parallel chunks and then recombined with near perfect efficiently. Hasn't that pretty much always been the case? One thing we are seeing more and more of is composable fabrics where the PCIe bus is effectively being extended outside the case such that you log into a single instance, and instead of seeing just 8 AIA's, you now see 32+. This makes the coordination a lot easier.
        refulgentis 2y ago
        I don't know how you keep rewording simple things everyone knows and are being patiently explained to you, as if you are encountering them the first time in the thread.
        latchkey 2y ago
        Which composable fabric am I talking about then?
        refulgentis 2y ago
        >> So it's only a net benefit when the work can be split up into tiny parallel chunks and then recombined with near perfect efficiently. > Hasn't that pretty much always been the case? I was talking about that, the fabric stuff is a non-sequitor, random hardware, doesn't make GPUs 2x in speed every 18 months, much less lead to a shift in everyday computing loads to GPUs. 1. Moore's Law also applies to GPUs. 2. If we could make use of 1000 cores for anything but long tail tasks, ye average CPU would have a lot more than 8 cores by now. That's what the million chickens thing is about. You can give me ultra-unobtianium fabric for "AIAs", it doesn't matter unless I have a algorithm that's massively parallelizable.

fragmede 2y ago

What's different today is the divide between training and inference. Inference is ridiculously cheap compared to training, and we're still early days with optimizations across the whole of the stack, so we'll have to see how it develops. Once constant training gets figured out, then we're really in for a ride.