4 ms·

GPT-3 is the same "tech" as GPT-2, with more training. GPT-2 is FOSS. I have a feeling that OpenAI's next architecture (if there ever is one) would still also b

by derefr 6y ago

GPT-3 is the same "tech" as GPT-2, with more training. GPT-2 is FOSS. I have a feeling that OpenAI's next architecture (if there ever is one) would still also be FOSS. I think OpenAI just chose a bad name for this for-profit initiative — "GPT-3" — that makes it sound like they were pivoting their company in a new direction with a new generation of tech. Really, GPT-3 should have been called something more like "GPT-2 Pro Plus Enterprise SaaS Edition." (Let's say "GPT-2++" for short.) Then it would have been clear that: 1. "GPT-2++" is not a generational leap over "GPT-2"; 2. an actual "GPT-3" would come later, and that it would be a new generation of tech; and 3. there would be a commercial "GPT-3++" to go along with "GPT-3", just like "GPT-2++" goes along with "GPT-2". (I can see why they called it GPT-3, though. Calling it "GPT-2++" probably wouldn't have made for very good news copy.)

marcosdumay 6y ago
GPT-2 Community and GPT-2 Enterprise. Those terms are so disseminated that I wouldn't be surprised if GPT-2 could suggest them.
- derefr 6y ago
  What I meant by my last statement is that no news outlet would have wanted to talk about "the innovative power of GPT-2 Enterprise." That just sounds fake, honestly. Every SaaS company wants to talk about the "innovative power" of the extra doodads they tack onto their Enterprise plans of their open-core product; where usually nobody is paying for their SaaS because of those doodads, but rather just because they want the service, want the ops handled for them, and want enterprise support if it goes down. But, by marketing it as a new version of the tech, "GPT-3", OpenAI gave journalists something they could actually report on without feeling like they're just shoving a PR release down people's throats. "The new generation of the tech can do all these amazing things; it's a leap forward!" is news. Even though, in this case, it's only a "quantity has a quality all its own" kind of "generational leap."
armoredkitten 6y ago
You make it sound as if GPT-3 is just the same GPT-2 model with some extra Enterprise-y features thrown in. They're completely different models, trained on different data, and vastly different sizes. GPT-2 had 1.5B parameters, and GPT-3 has 175B. It's two orders of magnitude larger. Sure, both models are using the same structures (attention layers, mostly), so it's a quantitative change rather than a qualitative change. But there's still a hell of a big difference between the two.
- derefr 6y ago
  Right, but GPT-2 was the name of the particular ML architecture they were studying the properties of; not the name of any specific model trained on that architecture. There was a pre-trained GPT-2 model offered for download. The whole "interesting thing" they were publishing about, was that models trained under the GPT-2 ML architecture were uniquely-good at transfer learning, and so any pre-trained GPT-2 model of sufficient size, would be extremely useful as a "seed" for doing your own model training on top of. They built one such model, but that model was not, itself, "GPT-2." Keep in mind, the training data for that model is open; you can download it yourself and reproduce the offered base-model from it if you like. That's because GPT-2 (the architecture) was formal academic computer science: journal papers and all. The particular pre-trained model, and its input training data, were just published as experimental data. It is under that lens, that I call GPT-3 "GPT-2++." It's a different model, but it's the same science. The model was never OpenAI's "product." The science itself was/is. Certainly, the SaaS pre-trained model named "GPT-3" is qualitatively different than the downloadable pre-trained base-model people refer to as "GPT-2." But so are all the various trained models people have built by training GPT-2 the architecture with their own inputs. The whole class of things trained on that architecture are fundamentally all "GPT-2 models." And so "GPT-3" is just one such "GPT-2 model." Just a really big, surprisingly-useful one.
  - ritter2a 6y ago
    > Right, but GPT-2 was the name of the particular ML architecture they were studying the properties of; not the name of any specific model trained on that architecture. That sounds like it would have been a reasonable choice for naming their research, but isn't the abbreviation "GPT" short for "Generative Pre-trained Transformer"? Seems like they very specifically refer to the pre-trained model, which I would also take from the GPT-2 paper's abstract: "Our largest model, GPT-2, is a 1.5B parameter Transformer[...]" [1] --- [1] https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf https://cdn.openai.com/better-language-models/language_model...

by derefr 6y ago

marcosdumay 6y ago

GPT-2 Community and GPT-2 Enterprise. Those terms are so disseminated that I wouldn't be surprised if GPT-2 could suggest them.

derefr 6y ago
What I meant by my last statement is that no news outlet would have wanted to talk about "the innovative power of GPT-2 Enterprise." That just sounds fake, honestly. Every SaaS company wants to talk about the "innovative power" of the extra doodads they tack onto their Enterprise plans of their open-core product; where usually nobody is paying for their SaaS because of those doodads, but rather just because they want the service, want the ops handled for them, and want enterprise support if it goes down. But, by marketing it as a new version of the tech, "GPT-3", OpenAI gave journalists something they could actually report on without feeling like they're just shoving a PR release down people's throats. "The new generation of the tech can do all these amazing things; it's a leap forward!" is news. Even though, in this case, it's only a "quantity has a quality all its own" kind of "generational leap."

armoredkitten 6y ago

You make it sound as if GPT-3 is just the same GPT-2 model with some extra Enterprise-y features thrown in. They're completely different models, trained on different data, and vastly different sizes. GPT-2 had 1.5B parameters, and GPT-3 has 175B. It's two orders of magnitude larger. Sure, both models are using the same structures (attention layers, mostly), so it's a quantitative change rather than a qualitative change. But there's still a hell of a big difference between the two.

derefr 6y ago
Right, but GPT-2 was the name of the particular ML architecture they were studying the properties of; not the name of any specific model trained on that architecture. There was a pre-trained GPT-2 model offered for download. The whole "interesting thing" they were publishing about, was that models trained under the GPT-2 ML architecture were uniquely-good at transfer learning, and so any pre-trained GPT-2 model of sufficient size, would be extremely useful as a "seed" for doing your own model training on top of. They built one such model, but that model was not, itself, "GPT-2." Keep in mind, the training data for that model is open; you can download it yourself and reproduce the offered base-model from it if you like. That's because GPT-2 (the architecture) was formal academic computer science: journal papers and all. The particular pre-trained model, and its input training data, were just published as experimental data. It is under that lens, that I call GPT-3 "GPT-2++." It's a different model, but it's the same science. The model was never OpenAI's "product." The science itself was/is. Certainly, the SaaS pre-trained model named "GPT-3" is qualitatively different than the downloadable pre-trained base-model people refer to as "GPT-2." But so are all the various trained models people have built by training GPT-2 the architecture with their own inputs. The whole class of things trained on that architecture are fundamentally all "GPT-2 models." And so "GPT-3" is just one such "GPT-2 model." Just a really big, surprisingly-useful one.
- ritter2a 6y ago
  > Right, but GPT-2 was the name of the particular ML architecture they were studying the properties of; not the name of any specific model trained on that architecture. That sounds like it would have been a reasonable choice for naming their research, but isn't the abbreviation "GPT" short for "Generative Pre-trained Transformer"? Seems like they very specifically refer to the pre-trained model, which I would also take from the GPT-2 paper's abstract: "Our largest model, GPT-2, is a 1.5B parameter Transformer[...]" [1] --- [1] https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf https://cdn.openai.com/better-language-models/language_model...