3 ms·

Even though none of us could actually afford to train it?

croemer 2mo ago
Some of us might be rich, or get funding, or in the future it could be useful when training is cheaper. Open source has benefits even if you can run yourself. You can read the code for understanding/insights. Other labs could replicate/build on it.
- gpugreg 2mo ago
  > Some of us might be rich I sure wish I had a few 100M of disposable income to train a frontier model. > or in the future it could be useful when training is cheaper. I do not think that physics will allow hardware getting that much faster. But maybe we will have different, cheaper architectures by then.
  - Muromec 2mo ago
    If you would tell people at the start of 20th century how much energy we consume, they may not believe you or think it is wasteful.
    - gpugreg 2mo ago
      Looks like global energy consumption has risen by an order of magnitude from 1900 to 2000: https://www.encyclopedie-energie.org/en/world-energy-consumption-1800-2000-results/#h3-1 https://www.encyclopedie-energie.org/en/world-energy-consump... Unfortunately, electricity prices did not fall by the same factor, so I fear that training a frontier model will still cause a an unsustainable dent in my monthly budget.
      - croemer 2mo ago
        Prices have dropped by a factor of around 6 in real terms since 1925 for US household electricity. https://chatgpt.com/share/6a6f3f3b-7624-83ed-a11a-248cb39728d2 https://chatgpt.com/share/6a6f3f3b-7624-83ed-a11a-248cb39728...
      - Muromec 2mo ago
        Only an order of magnitude? I'm both surprised by both the fact it's so low and the fact 1900 is 3x of 1800.
  - halJordan 2mo ago
    You say that but for like 2 years there was a guy on huggingface releasing quants as TheBloke. No identity no nothing except he likely had a grant or a university job that let him do it. Quants aren't the same as training but it was beyond 99% of people at the time (as you say this is) Cant stand when you guys try and force something as impossible on the rest of us simply because you could never accomplish it.
    - gpugreg 2mo ago
      I did not say that it is impossible. I just think that we need architectural improvements, or maybe even a fundamentally different approach to get something like Kimi K3 for cheap. The point I was trying to make was that we shouldn't just laze about and hope that hardware improvements will get us there. (Also, I know Tom Jobbins (TheBloke), and have personally contributed to increase the adoption of GGUF, e.g. in the transformers and ktransformers libraries, so I find the personal dig quite amusing.)
eptcyka 2mo ago
Maybe not now, but what about 10 years down the line?
croes 2mo ago
Yet, I couldn’t afford a PC to run the original Crysis when it came out, now the hardware isn’t an issue anymore
solarkraft 2mo ago
Yes.

Some of us might be rich, or get funding, or in the future it could be useful when training is cheaper. Open source has benefits even if you can run yourself. You can read the code for understanding/insights. Other labs could replicate/build on it.

gpugreg 2mo ago
> Some of us might be rich I sure wish I had a few 100M of disposable income to train a frontier model. > or in the future it could be useful when training is cheaper. I do not think that physics will allow hardware getting that much faster. But maybe we will have different, cheaper architectures by then.
- Muromec 2mo ago
  If you would tell people at the start of 20th century how much energy we consume, they may not believe you or think it is wasteful.
  - gpugreg 2mo ago
    Looks like global energy consumption has risen by an order of magnitude from 1900 to 2000: https://www.encyclopedie-energie.org/en/world-energy-consumption-1800-2000-results/#h3-1 https://www.encyclopedie-energie.org/en/world-energy-consump... Unfortunately, electricity prices did not fall by the same factor, so I fear that training a frontier model will still cause a an unsustainable dent in my monthly budget.
    - croemer 2mo ago
      Prices have dropped by a factor of around 6 in real terms since 1925 for US household electricity. https://chatgpt.com/share/6a6f3f3b-7624-83ed-a11a-248cb39728d2 https://chatgpt.com/share/6a6f3f3b-7624-83ed-a11a-248cb39728...
    - Muromec 2mo ago
      Only an order of magnitude? I'm both surprised by both the fact it's so low and the fact 1900 is 3x of 1800.
- halJordan 2mo ago
  You say that but for like 2 years there was a guy on huggingface releasing quants as TheBloke. No identity no nothing except he likely had a grant or a university job that let him do it. Quants aren't the same as training but it was beyond 99% of people at the time (as you say this is) Cant stand when you guys try and force something as impossible on the rest of us simply because you could never accomplish it.
  - gpugreg 2mo ago
    I did not say that it is impossible. I just think that we need architectural improvements, or maybe even a fundamentally different approach to get something like Kimi K3 for cheap. The point I was trying to make was that we shouldn't just laze about and hope that hardware improvements will get us there. (Also, I know Tom Jobbins (TheBloke), and have personally contributed to increase the adoption of GGUF, e.g. in the transformers and ktransformers libraries, so I find the personal dig quite amusing.)

eptcyka 2mo ago

Maybe not now, but what about 10 years down the line?

croes 2mo ago

Yet, I couldn’t afford a PC to run the original Crysis when it came out, now the hardware isn’t an issue anymore

solarkraft 2mo ago

Yes.