6 ms·

Does this mean you can train efficiently without GPUs? Presumably there will be a lot of interest.

crazygringo 2y ago
No. But it does potentially mean that either current or future-tweaked GPUs could run a lot more efficiently -- meaning much faster or with much less energy consumption. You still need the GPU parallelism though.
- fuzzfactor 2y ago
  I had a feeling it had to be something like massive waste due to a misguided feature of the algorithms that shouldn't have been there in the first place. Once the "math is done" quite likely it would have paid off better than most investments for the top people to have spent a few short years working with grossly underpowered hardware until they could come up with amazing results there before scaling up. Rather than grossly overpowered hardware before there was even deep understanding of the underlying processes. When you think about it, what we have seen from the latest ultra-high-powered "thinking" machines is truly so impressive. But if you are trying to fool somebody into believing that it's a real person it's still not "quite" there. Maybe a good benchmark would be to take a regular PC, and without reliance on AI just pull out all the stops and put all the effort into fakery itself. No holds barred, any trick you can think of. See what the electronics is capable of this way. There are some smart engineers, this would only take a few years but looks like it would have been a lot more affordable. Then with the same hardware if an AI alternative is not as convincing, something has got to be wrong. It's good to find out this type of thing before you go overboard. Regardless of speed or power, I never could have gotten an 8-bit computer to match the output of a 32-bit floating-point algorithm by using floating-point myself. Integers all the way and place the decimal where it's supposed to be when you're done. Once it's really figured out, how do you think it would feel being the one paying the electric bills up until now?
  - jimmaswell 2y ago
    Faster progress was absolutely worth it. Spending years agonizing over theory to save a bit of electric would have been a massive disservice to the world.
    - Forgeties79 2y ago
      “A bit”?
      - bartread 2y ago
        Yes, a large amount for - in the grand scheme of things - a short period of time (i.e., a quantity of energy usage in an intense spike that will be dwarfed by energy usage over time) can accurately be described as “a bit”. Of course, the impact is that AI will continue to become cheaper to use, and induced demand will continue the feedback loop driving the market as a result.
    - rossjudson 2y ago
      You're sort of presuming that LLMs are going to be a massive service to the world there, aren't you? I think the jury is still out on that one.
      - jimmaswell 2y ago
        They already have been. Even just in programming, even just Copilot has been a life changing productivity booster.
        wruza 2y ago
        Are you sure it’s a life changing productivity booster? Sometimes I look at my projects and wonder how would I explain it to an LLM what this code should have done if it didn’t exist yet. Must be a shitton of boilerplate programming for copilot to be a life-changing experience.
- 3abiton 2y ago
  This is still amazing work, imagine running chungus models on a single 3090.
  - Sohcahtoa82 2y ago
    The bottleneck on a consumer-grade GPU like a 3090 isn't the processing power, it's the lack of RAM. The PCI-Express bus ends up being your bottleneck from having to swap in parts of the model. Even with PCIe 5.0 and 16 lanes, you only get 64 GB/s of bandwidth. If you're trying to run a model too big for your GPU, then for every token, it has to reload the entire model. With a 70B parameter model, 8 bit quantization, you're looking at just under 1 token/sec just from having to transfer parts of the model in constantly. Making the actual computation faster won't make it any faster.
    - dragonwriter 2y ago
      OTOH, doesn't it also mean that (given appropriate software framework support) iGPUs with less processing capacity and slower-but-more RAM available (because system RAM is comparatively cheap and plentiful compared to VRAM) without swapping anything are more competitive against consumer dGPUs with fast-but-small RAM for both inference and training with larger models?
      - Sohcahtoa82 2y ago
        System memory isn't that fast, either. Even with DDR5-8400, the fastest memory you can get right now, you're only looking at a memory transfer speed of 67.2 GB/s, barely faster than the PCI-E bus. So even if you could store that entire 70B model in RAM, you're still getting just under 1 token/sec, and that's assuming your CPU doesn't become a bottleneck. Your best bet would likely be a laptop that has integrated system RAM with VRAM, but I don't think any of those offer enough RAM to store an entire 70B model. A 7B parameter model would work fine, but you could do those on a consumer-grade GPU anyways.
        DeveloperErrata 2y ago
pnt12 2y ago
The GPU main advantage is its parallelism - thousands of cores compared handful of cores in CPUs. If you're training models with billions of parameters, you're still gonna need that.

No. But it does potentially mean that either current or future-tweaked GPUs could run a lot more efficiently -- meaning much faster or with much less energy consumption. You still need the GPU parallelism though.

fuzzfactor 2y ago
I had a feeling it had to be something like massive waste due to a misguided feature of the algorithms that shouldn't have been there in the first place. Once the "math is done" quite likely it would have paid off better than most investments for the top people to have spent a few short years working with grossly underpowered hardware until they could come up with amazing results there before scaling up. Rather than grossly overpowered hardware before there was even deep understanding of the underlying processes. When you think about it, what we have seen from the latest ultra-high-powered "thinking" machines is truly so impressive. But if you are trying to fool somebody into believing that it's a real person it's still not "quite" there. Maybe a good benchmark would be to take a regular PC, and without reliance on AI just pull out all the stops and put all the effort into fakery itself. No holds barred, any trick you can think of. See what the electronics is capable of this way. There are some smart engineers, this would only take a few years but looks like it would have been a lot more affordable. Then with the same hardware if an AI alternative is not as convincing, something has got to be wrong. It's good to find out this type of thing before you go overboard. Regardless of speed or power, I never could have gotten an 8-bit computer to match the output of a 32-bit floating-point algorithm by using floating-point myself. Integers all the way and place the decimal where it's supposed to be when you're done. Once it's really figured out, how do you think it would feel being the one paying the electric bills up until now?
- jimmaswell 2y ago
  Faster progress was absolutely worth it. Spending years agonizing over theory to save a bit of electric would have been a massive disservice to the world.
  - Forgeties79 2y ago
    “A bit”?
    - bartread 2y ago
      Yes, a large amount for - in the grand scheme of things - a short period of time (i.e., a quantity of energy usage in an intense spike that will be dwarfed by energy usage over time) can accurately be described as “a bit”. Of course, the impact is that AI will continue to become cheaper to use, and induced demand will continue the feedback loop driving the market as a result.
  - rossjudson 2y ago
    You're sort of presuming that LLMs are going to be a massive service to the world there, aren't you? I think the jury is still out on that one.
    - jimmaswell 2y ago
      They already have been. Even just in programming, even just Copilot has been a life changing productivity booster.
      - wruza 2y ago
        Are you sure it’s a life changing productivity booster? Sometimes I look at my projects and wonder how would I explain it to an LLM what this code should have done if it didn’t exist yet. Must be a shitton of boilerplate programming for copilot to be a life-changing experience.
3abiton 2y ago
This is still amazing work, imagine running chungus models on a single 3090.
- Sohcahtoa82 2y ago
  The bottleneck on a consumer-grade GPU like a 3090 isn't the processing power, it's the lack of RAM. The PCI-Express bus ends up being your bottleneck from having to swap in parts of the model. Even with PCIe 5.0 and 16 lanes, you only get 64 GB/s of bandwidth. If you're trying to run a model too big for your GPU, then for every token, it has to reload the entire model. With a 70B parameter model, 8 bit quantization, you're looking at just under 1 token/sec just from having to transfer parts of the model in constantly. Making the actual computation faster won't make it any faster.
  - dragonwriter 2y ago
    OTOH, doesn't it also mean that (given appropriate software framework support) iGPUs with less processing capacity and slower-but-more RAM available (because system RAM is comparatively cheap and plentiful compared to VRAM) without swapping anything are more competitive against consumer dGPUs with fast-but-small RAM for both inference and training with larger models?
    - Sohcahtoa82 2y ago
      System memory isn't that fast, either. Even with DDR5-8400, the fastest memory you can get right now, you're only looking at a memory transfer speed of 67.2 GB/s, barely faster than the PCI-E bus. So even if you could store that entire 70B model in RAM, you're still getting just under 1 token/sec, and that's assuming your CPU doesn't become a bottleneck. Your best bet would likely be a laptop that has integrated system RAM with VRAM, but I don't think any of those offer enough RAM to store an entire 70B model. A 7B parameter model would work fine, but you could do those on a consumer-grade GPU anyways.
      - DeveloperErrata 2y ago

pnt12 2y ago

The GPU main advantage is its parallelism - thousands of cores compared handful of cores in CPUs. If you're training models with billions of parameters, you're still gonna need that.