5 ms·

I really can't stand the bait and switch branding of OpenAI. They are anything but open, and there is no reason to believe their tall tale that the only reason

I really can't stand the bait and switch branding of OpenAI. They are anything but open, and there is no reason to believe their tall tale that the only reason they don't release the model publicly is "ethical" reasons -- it's for monetary reasons and it's gatekeeping. Not only that, but they are playing a dangerous game strategically. Companies like Hugging Face actually release valuable parts of the open source NLP ecosystem which set new records for SOTA and which the rest of the industry can use. For all the practitioners here who probably know more about this than I do, help me understand: how much of OpenAI is smoke and mirrors? Is it just me, or is GPT-3 just GPT-2 but with way more training data? Are there fundamental conceptual differences with how the scale of that data is handled?

junon 6y ago
Pretty much share this sentiment. I read their FAQ page and it left a really bad taste in my mouth. A further question that I'd like to add - are there any _truly_ open source initiatives that can remotely compete with GPT-3? What's stopping the OSS community (specifically, not just "funding")? Is it lack of compute power? Lack of data?
- ForHackernews 6y ago
  I think it's just money. I'm not an expert, but my understanding is there's nothing fundamentally new or revolutionary about GPT-3. It's not a new AI paradigm, it's just taking existing techniques and turning them up to 11. If you throw absolutely enormous amounts of training data and compute resources at a bigass neural network, you get surprisingly good results. Anyone else with a few tens of millions of dollars could do similarly.
- est31 6y ago
  For GPT-3, I think they mainly scraped stuff off the internet. Anyone with enough time and machines can replicate that, but it is a non zero effort. GPT is an exception here I think. It's harder to build open source MRI diagnosis tools for example when nobody gives you open source MRI scans with diagnosis data. It's more a lack of compute power problem plus a lack of people problem. The compute cost alone was estimated to be 4.6 million USD. As for the people, folks who can push SOTA are extremely sought after. So the individuals involved also pay a large opportunity cost by working on things for free or comparatively small open source pay. Also, I think many open source organizations/communities haven't recognized the value of open source ML models yet and don't invest in them as much as in parametric models. I think this will come with time. Last, anyone with a computer can write code, while few people have access to the hardware needed.
- Voloskaya 6y ago
  Access to the right compute infrastructure. While the training itself cost 4.6$M (unsure if this include hyper parameter search done on smaller models beforehand), even if an open source organization had that money available, I don't believe they would be able to access the kind of infra needed to train GPT-3. The closest thing would probably be some TPU pods on GCP, but even then, I think the biggest pod you can publicly access is too small (unless you are ok training for 3 years).
  - webmaven 6y ago
    > even if an open source organization had that money available, I don't believe they would be able to access the kind of infra needed to train GPT-3. The closest thing would probably be some TPU pods on GCP, but even then, I think the biggest pod you can publicly access is too small (unless you are ok training for 3 years). Surely, at that budgetary scale, buying/building your own infrastructure would be cheaper than renting it? This is the same economic calculation large animation studios have to make WRT render farms (which also use similar types of hardware).
    - Voloskaya 6y ago
      The price of a cluster similar to the one OpenAI used is orders of magnitude higher. We are talking about a price tag that at the very minimum would be in the 150M+ range, you can probably multiply that price by 2 or 3 (10 thousand V100, 300k CPUs, infiniband between every node, network storage etc.). Almost no one has that kind of money to invest up front (especially not an hypothetical open source organization like discussed here). Even if somehow one had that money available, for this to be more economic than renting you need to have it running at full capacity at all times for years, not many research groups can come up with many interesting projects at a scale large enough to utilize that kind of cluster efficiently. So buying something like that is only conceivable for huge research groups with hundreds of researchers and tons of cash: Google Brain, DeepMind, Microsoft and probably Facebook. Even OpenAI wasn't able to afford it directly...
      - aquajet 6y ago
dustingetz 6y ago
OpenAI is exploiting the rules of society, as every human entity does under sufficiently strong optimization pressure. "Sufficiently Powerful Optimization Of Any Known Target Destroys All Value" https://thezvi.wordpress.com/2019/12/31/does-big-business-hate-your-family/ https://thezvi.wordpress.com/2019/12/31/does-big-business-ha... In the next 50 years, the digitization of everything will cause optimization pressure to steadily increase, we have an "unstoppable force meets immovable object" situation which historically results in societal collapse / war / "black swan" events

junon 6y ago

Pretty much share this sentiment. I read their FAQ page and it left a really bad taste in my mouth. A further question that I'd like to add - are there any _truly_ open source initiatives that can remotely compete with GPT-3? What's stopping the OSS community (specifically, not just "funding")? Is it lack of compute power? Lack of data?

ForHackernews 6y ago
I think it's just money. I'm not an expert, but my understanding is there's nothing fundamentally new or revolutionary about GPT-3. It's not a new AI paradigm, it's just taking existing techniques and turning them up to 11. If you throw absolutely enormous amounts of training data and compute resources at a bigass neural network, you get surprisingly good results. Anyone else with a few tens of millions of dollars could do similarly.
est31 6y ago
For GPT-3, I think they mainly scraped stuff off the internet. Anyone with enough time and machines can replicate that, but it is a non zero effort. GPT is an exception here I think. It's harder to build open source MRI diagnosis tools for example when nobody gives you open source MRI scans with diagnosis data. It's more a lack of compute power problem plus a lack of people problem. The compute cost alone was estimated to be 4.6 million USD. As for the people, folks who can push SOTA are extremely sought after. So the individuals involved also pay a large opportunity cost by working on things for free or comparatively small open source pay. Also, I think many open source organizations/communities haven't recognized the value of open source ML models yet and don't invest in them as much as in parametric models. I think this will come with time. Last, anyone with a computer can write code, while few people have access to the hardware needed.
Voloskaya 6y ago
Access to the right compute infrastructure. While the training itself cost 4.6$M (unsure if this include hyper parameter search done on smaller models beforehand), even if an open source organization had that money available, I don't believe they would be able to access the kind of infra needed to train GPT-3. The closest thing would probably be some TPU pods on GCP, but even then, I think the biggest pod you can publicly access is too small (unless you are ok training for 3 years).
- webmaven 6y ago
  > even if an open source organization had that money available, I don't believe they would be able to access the kind of infra needed to train GPT-3. The closest thing would probably be some TPU pods on GCP, but even then, I think the biggest pod you can publicly access is too small (unless you are ok training for 3 years). Surely, at that budgetary scale, buying/building your own infrastructure would be cheaper than renting it? This is the same economic calculation large animation studios have to make WRT render farms (which also use similar types of hardware).
  - Voloskaya 6y ago
    The price of a cluster similar to the one OpenAI used is orders of magnitude higher. We are talking about a price tag that at the very minimum would be in the 150M+ range, you can probably multiply that price by 2 or 3 (10 thousand V100, 300k CPUs, infiniband between every node, network storage etc.). Almost no one has that kind of money to invest up front (especially not an hypothetical open source organization like discussed here). Even if somehow one had that money available, for this to be more economic than renting you need to have it running at full capacity at all times for years, not many research groups can come up with many interesting projects at a scale large enough to utilize that kind of cluster efficiently. So buying something like that is only conceivable for huge research groups with hundreds of researchers and tons of cash: Google Brain, DeepMind, Microsoft and probably Facebook. Even OpenAI wasn't able to afford it directly...
    - aquajet 6y ago

dustingetz 6y ago

OpenAI is exploiting the rules of society, as every human entity does under sufficiently strong optimization pressure. "Sufficiently Powerful Optimization Of Any Known Target Destroys All Value" https://thezvi.wordpress.com/2019/12/31/does-big-business-hate-your-family/ https://thezvi.wordpress.com/2019/12/31/does-big-business-ha... In the next 50 years, the digitization of everything will cause optimization pressure to steadily increase, we have an "unstoppable force meets immovable object" situation which historically results in societal collapse / war / "black swan" events