5 ms·

> It’s interesting that these models are both massively expensive to produce and self-contained to a degree that you can distribute the end product in a torrent

> It’s interesting that these models are both massively expensive to produce and self-contained to a degree that you can distribute the end product in a torrent. I was trying to come to grasp with how much resource there is concentrated in one of these models. Somehow I come to the conclusion that it cost more than buying a jet airliner to train one of these models. And it is about the same order of money as commissioning and building a skyscraper in Manhattan. Is that correct approximately?

sva_ 4y ago
The crazy thing is that all these models are just one local minimum, out of a staggering (unknown?!) number of such points on the plane.
- basch 4y ago
  “Brute forcing a really inefficient approximation/estimator” is a good way to summarize it. It’s like having an overfit equation to a sample of data points, instead of the simpler actual line they fall near. They end up being black boxes, we have almost no idea how they work inside, and we have no idea how overtrained they are when something simpler could do the same thing.
  - kir-gadjello 4y ago
    Can "something simpler", for example, code correct function bodies from comments describing functions in natural language? I think people are too quick to dismiss the power of these models.
    - basch 4y ago
      I am by no means dismissing the power. They are created very chaotically, however. Spaghetti thrown at a wall. They are brute force approximations. They are wasteful. If LLaMa 13B is as powerful as previous 65B models, that's a significant amount of unnecessary paramaters lost/pruned in just this iterative upgrade alone. How small can they go? The fewest parameters that get the job done 99% as well is the way to go. There is also the difference between the rules and use of language being directly compressed into the model, vs all the information known to humans compressed into the model. A smaller model that ingests relevant information on the fly (more like Bing, that supplements itself with search), may be less wasteful and perform better. The current models being released are chosen because "they work" not because they are least fit and most performant optimized.
  - sva_ 4y ago
    I don't think the term "brute forcing" is an adequate term to describe gradient descent. Brute forcing would be to try all random weights with no system imo.
tedeh 4y ago
GPT-3 cost a few million dollars in compute to train from what I know.
sebzim4500 4y ago
No you are probably overestimating the cost by 1-2 orders of magnitude. GPT-3 probably cost under $5 million, and this model is smaller and there have been algorithmic improvements to training transformers since then.
- krisoft 4y ago
  > No you are probably overestimating the cost by 1-2 orders of magnitude. You are right! Wow. Thank you for correcting me. > GPT-3 probably cost under $5 million, Is that one training run or includes all the fiddling to find the right hyperparameters? Or there aren't many of those in these training or they are not that sensitive?
  - sebzim4500 4y ago
    I think they probably did a lot of hyperparameter searching to train the smaller models and then extrapolated for the largest model, but I'm just guessing. OpenAI had a finite amount of money when they were training GPT-3, they likely do it differently now that inference costs are significant compared to training costs.
- forgotOldLogin 4y ago
  In February 2018, OpenAI signed a two-year compute contract with Google that had a $63M minimum spend. See last page of their most recent available audited financials. https://rct.doj.ca.gov/Verification/Web/Download.aspx?saveas=560291.pdf&document_id=09027b8f803a8976 https://rct.doj.ca.gov/Verification/Web/Download.aspx?saveas...
  - dmix 4y ago
    So do they estimate how much computing power/time they will need and then find some upper tier minimum $ amount to get the maximum discount possible or getting a certain resource availability commitment from Google? That's an interesting accounting problem.
nodja 4y ago
For anyone curious, it took 2048 A100 GPUs to train LLaMa, each GPU costs roughly $15k, facebook probably gets some sort of discount. That's a $30Mil if you want to train at that scale. Also IIRC it took 23 days to train the biggest model. Someone else can do the power consumption cost calculations.
- fancyfredbot 4y ago
  $30m training cost is too high. Amazon's p4d.24xlarge is $32.77 an hour for 8 A100 GPUs. 2048 A100 GPUs for 23 days costs $4.6m at that rate. You might even get a discount.
  - orbz 4y ago
    At the same time I guarantee you they didn’t get it right the first time. I’m sure there were multiple (both serially and in parallel) runs as they worked out kinks and tuned hyper parameters.
    - zamnos 4y ago
      Not to mention, the kind of expertise to run this for a major corporation doesn't come for free either? Facebook employs quite a few high profile ML researchers who undoubtedly make mid-high six figure salaries.
  - nodja 4y ago
    15k is the price to buy a GPU, not to run it, you'd have to account for electricity costs which isn't so straight forward.
    - ithkuil 4y ago
      The point was that if you only need to train once, then it's cheaper to rent the GPUs than to buy them. If you need to train it multiple times, then the cost of buying the GPUs is amortized among runs. In any case the cost per run is going to be lower than 30m

sva_ 4y ago

The crazy thing is that all these models are just one local minimum, out of a staggering (unknown?!) number of such points on the plane.

basch 4y ago
“Brute forcing a really inefficient approximation/estimator” is a good way to summarize it. It’s like having an overfit equation to a sample of data points, instead of the simpler actual line they fall near. They end up being black boxes, we have almost no idea how they work inside, and we have no idea how overtrained they are when something simpler could do the same thing.
- kir-gadjello 4y ago
  Can "something simpler", for example, code correct function bodies from comments describing functions in natural language? I think people are too quick to dismiss the power of these models.
  - basch 4y ago
    I am by no means dismissing the power. They are created very chaotically, however. Spaghetti thrown at a wall. They are brute force approximations. They are wasteful. If LLaMa 13B is as powerful as previous 65B models, that's a significant amount of unnecessary paramaters lost/pruned in just this iterative upgrade alone. How small can they go? The fewest parameters that get the job done 99% as well is the way to go. There is also the difference between the rules and use of language being directly compressed into the model, vs all the information known to humans compressed into the model. A smaller model that ingests relevant information on the fly (more like Bing, that supplements itself with search), may be less wasteful and perform better. The current models being released are chosen because "they work" not because they are least fit and most performant optimized.
- sva_ 4y ago
  I don't think the term "brute forcing" is an adequate term to describe gradient descent. Brute forcing would be to try all random weights with no system imo.

tedeh 4y ago

GPT-3 cost a few million dollars in compute to train from what I know.

sebzim4500 4y ago

No you are probably overestimating the cost by 1-2 orders of magnitude. GPT-3 probably cost under $5 million, and this model is smaller and there have been algorithmic improvements to training transformers since then.

krisoft 4y ago
> No you are probably overestimating the cost by 1-2 orders of magnitude. You are right! Wow. Thank you for correcting me. > GPT-3 probably cost under $5 million, Is that one training run or includes all the fiddling to find the right hyperparameters? Or there aren't many of those in these training or they are not that sensitive?
- sebzim4500 4y ago
  I think they probably did a lot of hyperparameter searching to train the smaller models and then extrapolated for the largest model, but I'm just guessing. OpenAI had a finite amount of money when they were training GPT-3, they likely do it differently now that inference costs are significant compared to training costs.
forgotOldLogin 4y ago
In February 2018, OpenAI signed a two-year compute contract with Google that had a $63M minimum spend. See last page of their most recent available audited financials. https://rct.doj.ca.gov/Verification/Web/Download.aspx?saveas=560291.pdf&document_id=09027b8f803a8976 https://rct.doj.ca.gov/Verification/Web/Download.aspx?saveas...
- dmix 4y ago
  So do they estimate how much computing power/time they will need and then find some upper tier minimum $ amount to get the maximum discount possible or getting a certain resource availability commitment from Google? That's an interesting accounting problem.

nodja 4y ago

For anyone curious, it took 2048 A100 GPUs to train LLaMa, each GPU costs roughly $15k, facebook probably gets some sort of discount. That's a $30Mil if you want to train at that scale. Also IIRC it took 23 days to train the biggest model. Someone else can do the power consumption cost calculations.

fancyfredbot 4y ago
$30m training cost is too high. Amazon's p4d.24xlarge is $32.77 an hour for 8 A100 GPUs. 2048 A100 GPUs for 23 days costs $4.6m at that rate. You might even get a discount.
- orbz 4y ago
  At the same time I guarantee you they didn’t get it right the first time. I’m sure there were multiple (both serially and in parallel) runs as they worked out kinks and tuned hyper parameters.
  - zamnos 4y ago
    Not to mention, the kind of expertise to run this for a major corporation doesn't come for free either? Facebook employs quite a few high profile ML researchers who undoubtedly make mid-high six figure salaries.
- nodja 4y ago
  15k is the price to buy a GPU, not to run it, you'd have to account for electricity costs which isn't so straight forward.
  - ithkuil 4y ago
    The point was that if you only need to train once, then it's cheaper to rent the GPUs than to buy them. If you need to train it multiple times, then the cost of buying the GPUs is amortized among runs. In any case the cost per run is going to be lower than 30m