3 ms·

What's most surprising to me is that OpenAI really seems to believe that not publishing details will save them from competition. Everyone knows how these models

What's most surprising to me is that OpenAI really seems to believe that not publishing details will save them from competition. Everyone knows how these models work, and while I'm sure there is a bunch of "secret sauce" that OpenAI has built for training and fine-tuning, it's ridiculous to believe that the research community and competitors like Google and Facebook can't figure out the same. They just haven't really tried until recently because the capabilities and ROI of these models weren't obvious. No matter who you are, most of the smartest people work for someone else. The only competitive advantage that OpenAI has here is a headstart of 6-12 months from all the infrastructure investment into training these kinds of models. Now that everyone wants to build competing models with the same capabilities, this advantage is going to disappear very quickly.

newyankee 4y ago
What about the training data corpus ? Other than large cos like Google or Meta, can anyone else procure the same ?
- loveparade 4y ago
  Leaving the legal aspects of crawling aside, I think there is an important distinction here between 1. "can you procure it" and 2. "do you have enough money to process it all" 1. Yes, I think almost anyone can write code to procure the training corpus, in theory, and test it on a small scale 2. No, only the biggest labs and universities have enough resources to process such huge amounts of data and iterate on models with that scale. But that's just a matter of resources that can be overcome with partnerships between industry and academia that are common anyway. All the big labs already have huge efforts underway to reproduce GPT-X and it's just a matter of time before they catch up.
- gorbypark 4y ago
  At least as far as what the GPT-3 papers claimed, all (or most?) of the data used for training would be freely available for other competitors/researchers to acquire. Wikipedia, Common Crawl data, etc. I don’t believe OpenAI did their own crawling at all. With OpenAI not being really open, it’s hard to say for sure what exactly ended up in the training materials, though. GPT-4 is even more of a black box to anyone outside of OpenAI with very little information released on how it was trained.
tourist2d 4y ago
You say: OpenAI really seems to believe that not publishing details will save them from competition. Then say: The only competitive advantage that OpenAI has here is a headstart of 6-12 months It's almost as if they want to keep this advantage, huh? Blows my mind how business illiterate some HN commenters are.
- thwayunion 4y ago
  If someone doesn't file a Form 990, they are out for themselves and want to fuck (sorry... extract value from) everyone who isn't a (majority) share holder. Open AI is not the first for-profit philanthropy. It's just another evangelical church with a televangelist at the helm. TED talks are sermons. TBF: at least Open AI doesn't pretend to be a charity anymore... I feel sorry for the working sops who held MSFT stock in 401Ks and funded a massive tax write-off for the capital class. Dumbasses, amirite?
- loveparade 4y ago
  A headstart doesn't matter unless you can keep it. The point is that there are many mort smart people and resources outside of OpenAI and there are inside of OpenAI. If they focus their efforts, they will easily catch up. A headstart is not a competitive moat like network effects are. Go and try to raise money for your startup from a VC and tell them "well, everyone is doing the same as us, but we started 6 months earlier!!" - nobody cares.
  - fullshark 4y ago
    So OpenAI should just give up and release all their trade secrets? To what purpose?
    - Matl 4y ago
      I dunno, maybe the Open part of OpenAI should hint at it. The problem people are having is that OpenAI marketed themselves as supposedly democratizing AI, but it does the opposite.
sillysaurusx 4y ago
Au contraire, no one knows how large GPT-4 is, which is the single best predictor of performance (for a model trained to convergence). The GPT-4 paper spent much of its time writing about this — they did some small scale experiments with 1/1000th the compute, then picked a loss level they wanted and trained GPT-4 till it got it. Neither the exact loss level nor the number of parameters are revealed by the paper. Unfortunately it’s not possible to guess these from outside observations. Will this save them from competition? No, but it certainly makes things harder. Everyone immediately aimed at 175B the moment GPT-3 was published. GPT-4 is now a question mark.
- teruakohatu 4y ago
  For all we know they have hit 500B parameters with some clever unpublished optimisation, which would both give them an edge and if revealed would put a damper on the preveiling belief that LLMs can scale and scale (eg. 3x more params for less than 3x performance). As you say, there is absolutely no way for us to find out.
- whatshisface 4y ago
  I can't believe anyone considers a single number, which would work about equally well if it were 10% higher or lower, to be a trade secret.
  - typest 4y ago
    This is not really true. The Chinchilla paper showed that a 4% difference in loss between Chinchilla and Gopher led Chinchilla to blow Gopher out of the water at most tasks, including 30x performance in physics. Empirically, LLMs have shown to have emergent abilities appear at different loss levels. So, a 10% difference could really matter.
    - whatshisface 4y ago
      That ten percent is not loss it is parameter count.
  - 4y ago
qup 4y ago
Then what's everyone so mad about?

newyankee 4y ago

What about the training data corpus ? Other than large cos like Google or Meta, can anyone else procure the same ?

loveparade 4y ago
Leaving the legal aspects of crawling aside, I think there is an important distinction here between 1. "can you procure it" and 2. "do you have enough money to process it all" 1. Yes, I think almost anyone can write code to procure the training corpus, in theory, and test it on a small scale 2. No, only the biggest labs and universities have enough resources to process such huge amounts of data and iterate on models with that scale. But that's just a matter of resources that can be overcome with partnerships between industry and academia that are common anyway. All the big labs already have huge efforts underway to reproduce GPT-X and it's just a matter of time before they catch up.
gorbypark 4y ago
At least as far as what the GPT-3 papers claimed, all (or most?) of the data used for training would be freely available for other competitors/researchers to acquire. Wikipedia, Common Crawl data, etc. I don’t believe OpenAI did their own crawling at all. With OpenAI not being really open, it’s hard to say for sure what exactly ended up in the training materials, though. GPT-4 is even more of a black box to anyone outside of OpenAI with very little information released on how it was trained.

tourist2d 4y ago

You say: OpenAI really seems to believe that not publishing details will save them from competition. Then say: The only competitive advantage that OpenAI has here is a headstart of 6-12 months It's almost as if they want to keep this advantage, huh? Blows my mind how business illiterate some HN commenters are.

thwayunion 4y ago
If someone doesn't file a Form 990, they are out for themselves and want to fuck (sorry... extract value from) everyone who isn't a (majority) share holder. Open AI is not the first for-profit philanthropy. It's just another evangelical church with a televangelist at the helm. TED talks are sermons. TBF: at least Open AI doesn't pretend to be a charity anymore... I feel sorry for the working sops who held MSFT stock in 401Ks and funded a massive tax write-off for the capital class. Dumbasses, amirite?
loveparade 4y ago
A headstart doesn't matter unless you can keep it. The point is that there are many mort smart people and resources outside of OpenAI and there are inside of OpenAI. If they focus their efforts, they will easily catch up. A headstart is not a competitive moat like network effects are. Go and try to raise money for your startup from a VC and tell them "well, everyone is doing the same as us, but we started 6 months earlier!!" - nobody cares.
- fullshark 4y ago
  So OpenAI should just give up and release all their trade secrets? To what purpose?
  - Matl 4y ago
    I dunno, maybe the Open part of OpenAI should hint at it. The problem people are having is that OpenAI marketed themselves as supposedly democratizing AI, but it does the opposite.

sillysaurusx 4y ago

Au contraire, no one knows how large GPT-4 is, which is the single best predictor of performance (for a model trained to convergence). The GPT-4 paper spent much of its time writing about this — they did some small scale experiments with 1/1000th the compute, then picked a loss level they wanted and trained GPT-4 till it got it. Neither the exact loss level nor the number of parameters are revealed by the paper. Unfortunately it’s not possible to guess these from outside observations. Will this save them from competition? No, but it certainly makes things harder. Everyone immediately aimed at 175B the moment GPT-3 was published. GPT-4 is now a question mark.

teruakohatu 4y ago
For all we know they have hit 500B parameters with some clever unpublished optimisation, which would both give them an edge and if revealed would put a damper on the preveiling belief that LLMs can scale and scale (eg. 3x more params for less than 3x performance). As you say, there is absolutely no way for us to find out.
whatshisface 4y ago
I can't believe anyone considers a single number, which would work about equally well if it were 10% higher or lower, to be a trade secret.
- typest 4y ago
  This is not really true. The Chinchilla paper showed that a 4% difference in loss between Chinchilla and Gopher led Chinchilla to blow Gopher out of the water at most tasks, including 30x performance in physics. Empirically, LLMs have shown to have emergent abilities appear at different loss levels. So, a 10% difference could really matter.
  - whatshisface 4y ago
    That ten percent is not loss it is parameter count.
- 4y ago

qup 4y ago

Then what's everyone so mad about?