2 ms·

1) The costs will go down over time, much of the cost is the margin of NVIDIA and training new models 2) Absolutely. Thats like one hour of an engineer salary

by JanSt 2y ago

1) The costs will go down over time, much of the cost is the margin of NVIDIA and training new models 2) Absolutely. Thats like one hour of an engineer salary for a whole month.

sofixa 2y ago
> The costs will go down over time, much of the cost is the margin of NVIDIA and training new models Isn't each new model bigger and heavier and thus requries more compute to train?
- JanSt 2y ago
  Yes, but 1) you only need to train the model once and the inference is way cheaper. Train one great model (i.e. Claude 3.5) and you can get much more than $80/month worth out of it. 2) the hardware is getting much better and prices will fall drastically once there is a bit of a saturation of the market or another company starts putting out hardware that can compete with NVIDIA
  - sofixa 2y ago
    > Train one great model (i.e. Claude 3.5) and you can get much more than $80/month worth out of it Until the competition outcompetes you with their new model and you have to train a new superior one, because you have no moat. Which happens what, around every month or two? > the hardware is getting much better and prices will fall drastically once there is a bit of a saturation of the market or another company starts putting out hardware that can compete with NVIDIA Where is the hardware that can compete with NVIDIA going to come from? And if they don't have competition, which they don't, why would they bring down prices?
    - JanSt 2y ago
      The point is not that every lab will be profitable. There only needs to be one model in the end to increase our productivity massively, which is the point I'm making. Huge margins lead to a lot of competition trying to catch up, which is what makes market economies so successful.
    - ben_w 2y ago
      > Until the competition outcompetes you with their new model and you have to train a new superior one, because you have no moat. Which happens what, around every month or two? Eventually one of you runs out of money, but your customers keep getting better models until then; and if the loser in this race releases the weights on a suitable gratis license then your businesses can both lose. But that still leaves your customers with access to a model that's much cheaper to run than it was to create.
    - Workaccount2 2y ago
      Gemini models are trained and run on Google's in house TPU's, which frankly are incredible compared to H100's. In fact Claude was trained on TPUs. Google however does not sell these, you can only lease time on them via GCP.
- robrenaud 2y ago
  Then those new models get distilled into smaller ones. Raising the max intelligence of the models tends to raise the intelligence of all the models via distillation.

by JanSt 2y ago

1) The costs will go down over time, much of the cost is the margin of NVIDIA and training new models 2) Absolutely. Thats like one hour of an engineer salary for a whole month.

sofixa 2y ago

> The costs will go down over time, much of the cost is the margin of NVIDIA and training new models Isn't each new model bigger and heavier and thus requries more compute to train?

JanSt 2y ago
Yes, but 1) you only need to train the model once and the inference is way cheaper. Train one great model (i.e. Claude 3.5) and you can get much more than $80/month worth out of it. 2) the hardware is getting much better and prices will fall drastically once there is a bit of a saturation of the market or another company starts putting out hardware that can compete with NVIDIA
- sofixa 2y ago
  > Train one great model (i.e. Claude 3.5) and you can get much more than $80/month worth out of it Until the competition outcompetes you with their new model and you have to train a new superior one, because you have no moat. Which happens what, around every month or two? > the hardware is getting much better and prices will fall drastically once there is a bit of a saturation of the market or another company starts putting out hardware that can compete with NVIDIA Where is the hardware that can compete with NVIDIA going to come from? And if they don't have competition, which they don't, why would they bring down prices?
  - JanSt 2y ago
    The point is not that every lab will be profitable. There only needs to be one model in the end to increase our productivity massively, which is the point I'm making. Huge margins lead to a lot of competition trying to catch up, which is what makes market economies so successful.
  - ben_w 2y ago
    > Until the competition outcompetes you with their new model and you have to train a new superior one, because you have no moat. Which happens what, around every month or two? Eventually one of you runs out of money, but your customers keep getting better models until then; and if the loser in this race releases the weights on a suitable gratis license then your businesses can both lose. But that still leaves your customers with access to a model that's much cheaper to run than it was to create.
  - Workaccount2 2y ago
    Gemini models are trained and run on Google's in house TPU's, which frankly are incredible compared to H100's. In fact Claude was trained on TPUs. Google however does not sell these, you can only lease time on them via GCP.
robrenaud 2y ago
Then those new models get distilled into smaller ones. Raising the max intelligence of the models tends to raise the intelligence of all the models via distillation.