4 ms·

Nope nope nope. Any time we find more efficiency, we can trade it for more quality by doing more compute. We'll always use as much compute as we can afford, un

Nope nope nope. Any time we find more efficiency, we can trade it for more quality by doing more compute. We'll always use as much compute as we can afford, until we stop getting quality gains that are worth the added cost.

XorNot 2y ago
Isn't this article just about an optimization though, sans the title? I don't much care for all the "oh but the energy usage" claims in most tech things: it's all electricity, and it's all fungible. It usually seems to roll out as a proxy for "I don't like this thing". Like even with cryptocurrency, there were a lot of people mistaking the issue of scalability - namely that "as a store of value" crypto would consume incredible amounts of other resources (and a lot of people got stuck trying to figure out how somehow "a hash" could be reclaimed for useful resources) to do less then alternatives, with "the energy usage itself is the problem". Finding optimizations for LLMs is good because it means we can build cheaper LLMs, which means we can build larger LLMs then we otherwise could for some given constraint, which means we can miniaturize (or in this case specialize) more capable hardware. The thing which really matter is, can the energy usage be meaningfully limited to a sensible scaling factor given the capability that makes them useful? Because environmentally, I can install solar panels to do zero-carbon training (and if LLMs are as valuable as they're currently being priced, this is a no-brainer - if people aren't lying about solar being "cheaper then fossil fuels").
- alisonkisk 2y ago
  No, because the article is about sacrificing quality by using less energy. But the quality is already bad for non-toy purposes, so there is plenty of demand for higher-energy higher-quality systems.
- dartos 2y ago
  It’s good that we can build cheaper LLMs, but the problem companies guzzling energy for LLM training won’t use less energy, they’ll just have better models
  - kaba0 2y ago
    That energy is still on the order of a household’s yearly electricity, not that of Argentina as cryptos were, and that’s just for training. Inferring is much cheaper and arguably provides quite a lot of value (even though I also think it is overhyped), for very little energy consumption, probably more is lost due to inefficiency for any physical product.
    - dartos 2y ago
      I got news for you. The 1M GPUs that meta purchased running at full (or less probably) load 24/7 is more than the energy of a single household. The energy cost is in training, not inference.
      - kaba0 2y ago
        Which is the same order of magnitude? Also, how often do they train from scratch?
        dartos 2y ago
        I don’t understand your first question, sorry. But for the second: Llama 1, 2, and 3 all have different architectures and needed to be trained from scratch. Llama 1 was released February 2023. Same training story for openAI’s Sora, dalle, and 4o. All of mistral’s models Mamba, Kan, and Each version of rwkv (they’re on 6 now) Not that this list is a result of survivor bias. It’s only looking at their published models too. Not the probably 1000s of individual training experiments that go into producing each model.
        kaba0 2y ago
        Which is still absolutely nothing compared to something like youtube’s servers, which is absolutely nothing compared to something like the food industry. Like, if a couple of millions of people can use chatgpt in the manner they do today, would it matter if a house’s yearly energy budget was used up for that? Or 10?
- eru 2y ago
  > Like even with cryptocurrency, there were a lot of people mistaking the issue of scalability - namely that "as a store of value" crypto would consume incredible amounts of other resources (and a lot of people got stuck trying to figure out how somehow "a hash" could be reclaimed for useful resources) to do less then alternatives, with "the energy usage itself is the problem". To be fair, technology-wise they mostly solved this problem via proof-of-stake. From an individual point of view you still expense enormous resources as a miner / validator in a proof-of-stake system. It's just that now the resources come in the form of lost opportunity costs for your staked tokens (eg staked Ethereum). But from aggregated perspective of society, staked Ethereum is essentially free. That has some parallels to how acquiring regular money, like USD, is something individuals spend a lot of effort on. But for the whole of society, printing USD is essentially free. > Because environmentally, I can install solar panels to do zero-carbon training (and if LLMs are as valuable as they're currently being priced, this is a no-brainer - if people aren't lying about solar being "cheaper then fossil fuels"). There's still opportunity costs for that energy. Unless you have truly stranded electricity that couldn't be used for anything else. > Finding optimizations for LLMs is good because it means we can build cheaper LLMs, which means we can build larger LLMs then we otherwise could for some given constraint, which means we can miniaturize (or in this case specialize) more capable hardware. The thing which really matter is, can the energy usage be meaningfully limited to a sensible scaling factor given the capability that makes them useful? I agree with that paragraph. It's all about trade-offs. If we can shift the efficiency frontier, that's good. Then people can decide whether they want cheaper models at the same performance, or pay the same energy-price for better models, or a combination thereof. Or pay more energy for even better model
shikon7 2y ago
Yes, it seems the more efficient use we have for compute, the more valuable compute is to us, and the more compute we will be able to afford.
kazinator 2y ago
Cost is basically time here. "How much compute we can afford" is really "how long we are willing to wait for the result".
- trashtester 2y ago
  Then again, time * size_of_cluster = cost. If you train on a cluster that costs >$1M/day to operate, the wait time is likely to be a smaller concern than the financial cost, unless you're REALLY in a hurry to beat some competitor.

XorNot 2y ago

Isn't this article just about an optimization though, sans the title? I don't much care for all the "oh but the energy usage" claims in most tech things: it's all electricity, and it's all fungible. It usually seems to roll out as a proxy for "I don't like this thing". Like even with cryptocurrency, there were a lot of people mistaking the issue of scalability - namely that "as a store of value" crypto would consume incredible amounts of other resources (and a lot of people got stuck trying to figure out how somehow "a hash" could be reclaimed for useful resources) to do less then alternatives, with "the energy usage itself is the problem". Finding optimizations for LLMs is good because it means we can build cheaper LLMs, which means we can build larger LLMs then we otherwise could for some given constraint, which means we can miniaturize (or in this case specialize) more capable hardware. The thing which really matter is, can the energy usage be meaningfully limited to a sensible scaling factor given the capability that makes them useful? Because environmentally, I can install solar panels to do zero-carbon training (and if LLMs are as valuable as they're currently being priced, this is a no-brainer - if people aren't lying about solar being "cheaper then fossil fuels").

alisonkisk 2y ago
No, because the article is about sacrificing quality by using less energy. But the quality is already bad for non-toy purposes, so there is plenty of demand for higher-energy higher-quality systems.
dartos 2y ago
It’s good that we can build cheaper LLMs, but the problem companies guzzling energy for LLM training won’t use less energy, they’ll just have better models
- kaba0 2y ago
  That energy is still on the order of a household’s yearly electricity, not that of Argentina as cryptos were, and that’s just for training. Inferring is much cheaper and arguably provides quite a lot of value (even though I also think it is overhyped), for very little energy consumption, probably more is lost due to inefficiency for any physical product.
  - dartos 2y ago
    I got news for you. The 1M GPUs that meta purchased running at full (or less probably) load 24/7 is more than the energy of a single household. The energy cost is in training, not inference.
    - kaba0 2y ago
      Which is the same order of magnitude? Also, how often do they train from scratch?
      - dartos 2y ago
        I don’t understand your first question, sorry. But for the second: Llama 1, 2, and 3 all have different architectures and needed to be trained from scratch. Llama 1 was released February 2023. Same training story for openAI’s Sora, dalle, and 4o. All of mistral’s models Mamba, Kan, and Each version of rwkv (they’re on 6 now) Not that this list is a result of survivor bias. It’s only looking at their published models too. Not the probably 1000s of individual training experiments that go into producing each model.
        kaba0 2y ago
        Which is still absolutely nothing compared to something like youtube’s servers, which is absolutely nothing compared to something like the food industry. Like, if a couple of millions of people can use chatgpt in the manner they do today, would it matter if a house’s yearly energy budget was used up for that? Or 10?
eru 2y ago
> Like even with cryptocurrency, there were a lot of people mistaking the issue of scalability - namely that "as a store of value" crypto would consume incredible amounts of other resources (and a lot of people got stuck trying to figure out how somehow "a hash" could be reclaimed for useful resources) to do less then alternatives, with "the energy usage itself is the problem". To be fair, technology-wise they mostly solved this problem via proof-of-stake. From an individual point of view you still expense enormous resources as a miner / validator in a proof-of-stake system. It's just that now the resources come in the form of lost opportunity costs for your staked tokens (eg staked Ethereum). But from aggregated perspective of society, staked Ethereum is essentially free. That has some parallels to how acquiring regular money, like USD, is something individuals spend a lot of effort on. But for the whole of society, printing USD is essentially free. > Because environmentally, I can install solar panels to do zero-carbon training (and if LLMs are as valuable as they're currently being priced, this is a no-brainer - if people aren't lying about solar being "cheaper then fossil fuels"). There's still opportunity costs for that energy. Unless you have truly stranded electricity that couldn't be used for anything else. > Finding optimizations for LLMs is good because it means we can build cheaper LLMs, which means we can build larger LLMs then we otherwise could for some given constraint, which means we can miniaturize (or in this case specialize) more capable hardware. The thing which really matter is, can the energy usage be meaningfully limited to a sensible scaling factor given the capability that makes them useful? I agree with that paragraph. It's all about trade-offs. If we can shift the efficiency frontier, that's good. Then people can decide whether they want cheaper models at the same performance, or pay the same energy-price for better models, or a combination thereof. Or pay more energy for even better model

shikon7 2y ago

Yes, it seems the more efficient use we have for compute, the more valuable compute is to us, and the more compute we will be able to afford.

kazinator 2y ago

Cost is basically time here. "How much compute we can afford" is really "how long we are willing to wait for the result".

trashtester 2y ago
Then again, time * size_of_cluster = cost. If you train on a cluster that costs >$1M/day to operate, the wait time is likely to be a smaller concern than the financial cost, unless you're REALLY in a hurry to beat some competitor.