3 ms·

The hardware required for training foundational models is tremendously expensive, specialized, and needs to be colocated in the same facility. You can't do thi

The hardware required for training foundational models is tremendously expensive, specialized, and needs to be colocated in the same facility. You can't do this kind of math without these multi-million dollar facilities and the government licensing that will eventually be required to run one.

knaik94 3y ago
These foundational models have already been created. Government backing has already been used to create a fully open source model. The effort has been multi-national. Government licenses aren't currently required to run one. "An award of computer time was provided by the INCITE program. This research also used resources of the Oak Ridge Leadership Computing Facility (OLCF), which is a DOE Office of Science User Facility supported under Contract DE-AC05-00OR22725." Models: https://www.together.xyz/blog/redpajama-models-v1 https://www.together.xyz/blog/redpajama-models-v1 Training Data: https://www.together.xyz/blog/redpajama https://www.together.xyz/blog/redpajama
hellojesus 3y ago
They'll just build one in Iran or some other US advisary. The US will lose the arms race and the tax revenue.
nate_meurer 3y ago
Sam Altman estimated that the cost to train GPT-4 was about $100 million. Not only is this not tremendously expensive, it's pocket change for thousand of corporations around the world. But even if what you said was true, this technology is progressing faster than we can write news stories about it. OpenAI itself is only a few years old, and you can already run a GPT-3-class model on your laptop. Any prognostication you make about cost will be valid for approximately a month. And, I'm sorry I can't help myself, what the heck do you mean by colocated? Exactly what things need to be colocated?
- astrange 3y ago
  > Sam Altman estimated that the cost to train GPT-4 was about $100 million. Not only is this not tremendously expensive, it's pocket change for thousand of corporations around the world. I think he said it was more, and that was for a company already capable of creating GPT-3.5. The only company capable of it, in fact.
  - nate_meurer 3y ago
    The only company, huh? Strange that so many other companies also building LLMs. You better go tell them all not to waste their time, OpenAI has it locked up. I'm sure they'll be interested to hear your ideas.
    - astrange 3y ago
      Have you tried Google Bard? It's kind of bad.

knaik94 3y ago

These foundational models have already been created. Government backing has already been used to create a fully open source model. The effort has been multi-national. Government licenses aren't currently required to run one. "An award of computer time was provided by the INCITE program. This research also used resources of the Oak Ridge Leadership Computing Facility (OLCF), which is a DOE Office of Science User Facility supported under Contract DE-AC05-00OR22725." Models: https://www.together.xyz/blog/redpajama-models-v1 https://www.together.xyz/blog/redpajama-models-v1 Training Data: https://www.together.xyz/blog/redpajama https://www.together.xyz/blog/redpajama

hellojesus 3y ago

They'll just build one in Iran or some other US advisary. The US will lose the arms race and the tax revenue.

nate_meurer 3y ago

Sam Altman estimated that the cost to train GPT-4 was about $100 million. Not only is this not tremendously expensive, it's pocket change for thousand of corporations around the world. But even if what you said was true, this technology is progressing faster than we can write news stories about it. OpenAI itself is only a few years old, and you can already run a GPT-3-class model on your laptop. Any prognostication you make about cost will be valid for approximately a month. And, I'm sorry I can't help myself, what the heck do you mean by colocated? Exactly what things need to be colocated?

astrange 3y ago
> Sam Altman estimated that the cost to train GPT-4 was about $100 million. Not only is this not tremendously expensive, it's pocket change for thousand of corporations around the world. I think he said it was more, and that was for a company already capable of creating GPT-3.5. The only company capable of it, in fact.
- nate_meurer 3y ago
  The only company, huh? Strange that so many other companies also building LLMs. You better go tell them all not to waste their time, OpenAI has it locked up. I'm sure they'll be interested to hear your ideas.
  - astrange 3y ago
    Have you tried Google Bard? It's kind of bad.