5 ms·

From the article: "Of course, you need a sufficiently large model to be able to learn from all this data, which is why GPT-3 is 175 billion parameters and prob

From the article: "Of course, you need a sufficiently large model to be able to learn from all this data, which is why GPT-3 is 175 billion parameters and probably cost between $1m-10m in compute cost to train.[2]" So, perhaps better title would be "GPT in 60 Lines of Numpy (and $1m-$10m)"

MuffinFlavored 4y ago
Will one business model be for OpenAI to "license" out access to their trained model? How large is the model on disk(s) once it is trained?
- shagie 4y ago
  From various sources, the model itself is about 800 GB on disk.
- theptip 4y ago
  Perhaps I’m missing your point, but isn’t that what they do with their API right now? You pay for text completions, and can fine-tune their model with your data.
  - veqq 4y ago
    But you can't run the code on your own machine.
    - est 4y ago
      > But you can't run the code on your own machine. iirc GPT-3 itself alone is some 500TB in size. You need a really, really big machine to run LLMs, the first L means Large.
    - mattnewton 4y ago
      Of course, if they leaked the model weight’s and a local inference binary for it they would lose the ability to charge for it. Clones with the weights would crop up all over the place.
- hackernewds 4y ago
  They must have time traveled to today in the past and read your comment, since this is precisely their business model!
rvz 4y ago
And it will be even more expensive to train it again on larger amounts of data and with a model with 10 times more parameters. Only Big Tech giants like Microsoft, Google, etc can afford to foot the bill and throw away millions into training LLMs, whilst we celebrate and hype about ChatGPT and LLMs getting bigger and significantly more expensive to train when they get confused, hallucinate over silly inputs and confidently generate bullshit. That can't be a good thing. OpenAI's ClosedAI model needs to be disrupted like how Stable Diffusion challenged DALLE-2 with an open source AI model.
- Kranar 4y ago
  I disagree, I run a small tech company that has a group that's been experimenting with stable diffusion and we noticed that an extreme version of the Pareto Principle applies here as well where you can get ~90% of the benefits for like 5% of the cost, combined with the fact that computing power is continuously getting cheaper. Based on that groups success, they've recently proposed a mini project inspired by GPT that I am considering funding; the data its trained on is all publicly available for free, and most it comes from Common Crawl. I suspect that it will also yield similar results, where you can tailor your own version of GPT and get reasonably good models for a fraction of the price as well. We're no where close to the scale of Big Tech giants, but I've noticed for the better part of 15 years that small companies can actually derive a great deal of the benefits that larger companies have for a fraction of the cost if they play it smart and keep things tight.
  - 99_00 4y ago
    Do you think it is possible for the AI to request information to fill in gaps in it's model? For example, the AI doesn't have enough information about a companies process, or a regulation. It chats with an expert to fill in the gaps. I have no understanding of AI
    - TheCoreh 4y ago
      It is! You can specify on its prompt that it should "request additional info via search query, using the following syntax: [[search terms here]], before coming to a final conclusion" then you integrate it with a traditional knowledge base textual look up, and run it again with that information concatenated
99_00 4y ago
Anyone know what the minimum cost for creating a model is and what the limitation would be?
- sharemywin 4y ago
  this is pretty small: https://github.com/karpathy/nanoGPT https://github.com/karpathy/nanoGPT
- deleted 4y ago
  [deleted]
zeknife 4y ago
There are GPT-2 checkpoints small enough to run on basically any modern computer
pumanoir 4y ago
I saw this [1] presentation where they use scheme to train GPT on a single consumer GPU. I've had no luck finding the 'scorch' compiler they mentioned in the video. 1. https://youtu.be/rDke29MbKQA?list=PLyrlk8Xaylp7NvZ1r-eTIUHdyHQg0auvo&t=3035 https://youtu.be/rDke29MbKQA?list=PLyrlk8Xaylp7NvZ1r-eTIUHdy...

MuffinFlavored 4y ago

Will one business model be for OpenAI to "license" out access to their trained model? How large is the model on disk(s) once it is trained?

shagie 4y ago
From various sources, the model itself is about 800 GB on disk.
theptip 4y ago
Perhaps I’m missing your point, but isn’t that what they do with their API right now? You pay for text completions, and can fine-tune their model with your data.
- veqq 4y ago
  But you can't run the code on your own machine.
  - est 4y ago
    > But you can't run the code on your own machine. iirc GPT-3 itself alone is some 500TB in size. You need a really, really big machine to run LLMs, the first L means Large.
  - mattnewton 4y ago
    Of course, if they leaked the model weight’s and a local inference binary for it they would lose the ability to charge for it. Clones with the weights would crop up all over the place.
hackernewds 4y ago
They must have time traveled to today in the past and read your comment, since this is precisely their business model!

rvz 4y ago

And it will be even more expensive to train it again on larger amounts of data and with a model with 10 times more parameters. Only Big Tech giants like Microsoft, Google, etc can afford to foot the bill and throw away millions into training LLMs, whilst we celebrate and hype about ChatGPT and LLMs getting bigger and significantly more expensive to train when they get confused, hallucinate over silly inputs and confidently generate bullshit. That can't be a good thing. OpenAI's ClosedAI model needs to be disrupted like how Stable Diffusion challenged DALLE-2 with an open source AI model.

Kranar 4y ago
I disagree, I run a small tech company that has a group that's been experimenting with stable diffusion and we noticed that an extreme version of the Pareto Principle applies here as well where you can get ~90% of the benefits for like 5% of the cost, combined with the fact that computing power is continuously getting cheaper. Based on that groups success, they've recently proposed a mini project inspired by GPT that I am considering funding; the data its trained on is all publicly available for free, and most it comes from Common Crawl. I suspect that it will also yield similar results, where you can tailor your own version of GPT and get reasonably good models for a fraction of the price as well. We're no where close to the scale of Big Tech giants, but I've noticed for the better part of 15 years that small companies can actually derive a great deal of the benefits that larger companies have for a fraction of the cost if they play it smart and keep things tight.
- 99_00 4y ago
  Do you think it is possible for the AI to request information to fill in gaps in it's model? For example, the AI doesn't have enough information about a companies process, or a regulation. It chats with an expert to fill in the gaps. I have no understanding of AI
  - TheCoreh 4y ago
    It is! You can specify on its prompt that it should "request additional info via search query, using the following syntax: [[search terms here]], before coming to a final conclusion" then you integrate it with a traditional knowledge base textual look up, and run it again with that information concatenated

99_00 4y ago

Anyone know what the minimum cost for creating a model is and what the limitation would be?

sharemywin 4y ago
this is pretty small: https://github.com/karpathy/nanoGPT https://github.com/karpathy/nanoGPT
deleted 4y ago
[deleted]

zeknife 4y ago

There are GPT-2 checkpoints small enough to run on basically any modern computer

pumanoir 4y ago

I saw this [1] presentation where they use scheme to train GPT on a single consumer GPU. I've had no luck finding the 'scorch' compiler they mentioned in the video. 1. https://youtu.be/rDke29MbKQA?list=PLyrlk8Xaylp7NvZ1r-eTIUHdyHQg0auvo&t=3035 https://youtu.be/rDke29MbKQA?list=PLyrlk8Xaylp7NvZ1r-eTIUHdy...