10 ms·

OpenAI is giving Microsoft exclusive access to its GPT-3 language model

172 points by ColinWright 6y ago

jamesrom 6y ago
Read this: https://openai.com/charter/ https://openai.com/charter/ Then read this: https://openai.com/blog/openai-licenses-gpt-3-technology-to-microsoft/ https://openai.com/blog/openai-licenses-gpt-3-technology-to-... (the very first dot point) Dear OpenAI, please provide more details of the arrangement with Microsoft. The lack of transparency here shows a clear departure from your charter.
- amelius 6y ago
  Can someone who has done work for OpenAI sue them for not sticking with their charter?
- jonathathan 6y ago
  The charter only constrains their use of AGI, which they define at the top in a way that excludes all of their current and near-future work.
  - mkolodny 6y ago
    "We commit to use any influence we obtain over AGI’s deployment to ensure it is used for the benefit of all, and to avoid enabling uses of AI or AGI that harm humanity or unduly concentrate power." That sounds to me like they committed to avoid using any AI to concentrate power.
    - still_grokking 6y ago
      But it is written: "unduly" concentrate. The whole sentence is just yada-yada anyway. All "commitments" are constrained by some undefined or impossible exception. Defining "unduly" would be even considerably easy compared to finding a definition for what could be meant by "harm humanity" or be "for the benefit of all". The last constrain is nonsense in itself actually, or does "all" except Microsoft who would not benefit if they wouldn't got an exclusive contract? Not to mention all the people who will lose their jobs as those jobs will be replaced by AI eventually… They didn't say "all equally" which would make at least some little bit sense.
- cscurmudgeon 6y ago
  This is a very vexing situation. I applied for access months ago. I am a scientist and academic with dozens of articles on topics related to GPT (and especially survey articles and a few articles reasonably critical of some NLP/ML/DL frameworks). I am not sure I can reveal more without doxing myself. I haven't heard anything from them yet. They are free to not release GPT-3 but not free to mislead the open community with this open and holier-than-thou nonsense.
yowlingcat 6y ago
I really can't stand the bait and switch branding of OpenAI. They are anything but open, and there is no reason to believe their tall tale that the only reason they don't release the model publicly is "ethical" reasons -- it's for monetary reasons and it's gatekeeping. Not only that, but they are playing a dangerous game strategically. Companies like Hugging Face actually release valuable parts of the open source NLP ecosystem which set new records for SOTA and which the rest of the industry can use. For all the practitioners here who probably know more about this than I do, help me understand: how much of OpenAI is smoke and mirrors? Is it just me, or is GPT-3 just GPT-2 but with way more training data? Are there fundamental conceptual differences with how the scale of that data is handled?
- junon 6y ago
  Pretty much share this sentiment. I read their FAQ page and it left a really bad taste in my mouth. A further question that I'd like to add - are there any _truly_ open source initiatives that can remotely compete with GPT-3? What's stopping the OSS community (specifically, not just "funding")? Is it lack of compute power? Lack of data?
  - ForHackernews 6y ago
    I think it's just money. I'm not an expert, but my understanding is there's nothing fundamentally new or revolutionary about GPT-3. It's not a new AI paradigm, it's just taking existing techniques and turning them up to 11. If you throw absolutely enormous amounts of training data and compute resources at a bigass neural network, you get surprisingly good results. Anyone else with a few tens of millions of dollars could do similarly.
  - est31 6y ago
    For GPT-3, I think they mainly scraped stuff off the internet. Anyone with enough time and machines can replicate that, but it is a non zero effort. GPT is an exception here I think. It's harder to build open source MRI diagnosis tools for example when nobody gives you open source MRI scans with diagnosis data. It's more a lack of compute power problem plus a lack of people problem. The compute cost alone was estimated to be 4.6 million USD. As for the people, folks who can push SOTA are extremely sought after. So the individuals involved also pay a large opportunity cost by working on things for free or comparatively small open source pay. Also, I think many open source organizations/communities haven't recognized the value of open source ML models yet and don't invest in them as much as in parametric models. I think this will come with time. Last, anyone with a computer can write code, while few people have access to the hardware needed.
nitinreddy88 6y ago
Old news and already discussed https://news.ycombinator.com/item?id=24558329 https://news.ycombinator.com/item?id=24558329
ackbar03 6y ago
Well Microsoft did give them some sort of discount or something for their azure servers for training. I think it was part of the agreement or something
- webmaven 6y ago
  > Well Microsoft did give them some sort of discount or something for their azure servers for training. I think it was part of the agreement or something I think about half of the money Microsoft invested in OpenAI has been cash, the rest was in the form of Azure credits.
  - webmaven 6y ago
    I realize it isn't obvious why this probably precludes a discount. For any given deal, it is much simpler for all parties to adjust the amount of credits being given than to negotiate a discount.
jgalt212 6y ago
open != exclusive
TaupeRanger 6y ago
Thus changing its name to ExclusiveAI? No? Still the air of moral superiority?
- DaiPlusPlus 6y ago
  When Progress Inc. changed the name of their awful-yet-surprisingly-popular closed-source (even [anti-open-source](https://www.progress.com/tutorials/odbc/open-source-database https://www.progress.com/tutorials/odbc/open-source-database) ) 4GL from ProgressDB to "OpenEdge" it was a reminder to me that there is absolutely nothing to a name. "A rose by any other name would smell as sweet" - similarly: "Open[BLANK] by any other name would be just as much marketing bollocks".
  - PeterStuer 6y ago
    What does this have to do with the discussion about OpenAI changing not just strategy but what feels to many as changing their core values that they always promoted as being the central reason for them existing? "But what about ..." is irrelevant https://en.wikipedia.org/wiki/Whataboutism https://en.wikipedia.org/wiki/Whataboutism
    - DaiPlusPlus 6y ago
      > What does this have to do with the discussion about OpenAI changing not just strategy Nothing to do with that. Nor is this a case of whataboutism. I am not justifying one company's actions based on another unrelated company's poor business decisioins. I am stating that given this deal with Microsoft, the "Open" in OpenAI's name is now meaningless and/or dishonest. I used the example of Progress/OpenEdge as another example of a company putting "open" right into their name but failing to live up to their name - implying that Progress' name-change (and also: OpenAI's initial name) was a cynical marketing ploy to make people believe they're the _good guys_ in all this, especially when the AI/ML field has many detractors from the altruistic camps, especially after Deep-Fakes, Palantir, Cambridge Analytica, and Facebook have all left a bad taste in everyone's' mouth.
simonkafan 6y ago
I still don't see or have read about a useful use case for GPT-3. Maybe for procedurally generated chitchat between NPC in video games or an advanced Lorem ipsum generator but that's about it.
- bananaface 6y ago
  Auto-formatting punctuation - or syntax. You can show it a few examples and it will insert punctuation into raw words coming in from speech recognition, or convert badly formatted Python into well-formatted Python. It can correct syntax errors in C++ (!). More generally, it can convert text from a simple, unstructured form into more complex, more structured form, with just a few example transforms. It's astonishingly good at it, but I haven't seen much on the public net about it. I don't know why people aren't exploring it more tbh.
  - tensility 6y ago
    Now that it is a captive technology, the question becomes why should anyone want to explore it other than Microsoft.
    - bananaface 6y ago
      It's not captive. It's quite replicable (and impossible to monopolise), it's just expensive.
- TaupeRanger 6y ago
  Yes. People were literally claiming to be able to write code using English language descriptions. This guy on Twitter generated a ton of hype with these cherry picked examples...even from people who should know better like Eliezer Yudkowsky. https://twitter.com/sharifshameem/status/1284095222939451393?s=19 https://twitter.com/sharifshameem/status/1284095222939451393...
  - avoidboringppl 6y ago
    Also interesting to note that the hype came a while after the release of GPT-3 (the tweet was about a month after), and that many of the examples could have possibly been generated from GPT-2 (which didn't get as much hype). Put another way, you could show ppl GPT-2 examples from a year ago and claim they were from GPT-3, and I don't think many would know the difference. I do think GPT-3 has shown improvement, and it's a step forward, but it probably tells more about how humans interpret rather than how AI might work. Wrote about it more in the link below: https://avoidboringpeople.substack.com/p/doctor-gpt-3 https://avoidboringpeople.substack.com/p/doctor-gpt-3
chosen1111 6y ago
Lets just build another and be done with it. It’s expensive to make with current parameters. Can anyone weigh in on the following as a means of lowering compute cost. https://www.infoq.com/news/2020/10/training-exceeds-gpt3/ https://www.infoq.com/news/2020/10/training-exceeds-gpt3/
- Voloskaya 6y ago
  This thing is pure clickbait. It's better than GPT-3 on language representation tasks where GPT-3 was not very good to start with. GPT-3 is a language generation model, and on language generation tasks, GPT-3 is absolutely better than anything else right now. In PET's author own words: "GPT-3 certainly is much better than our approach at generating long sequences of text (e.g., summarization or machine translation)" There are things that can be done to have something similar to GPT-3 in performance at a lower cost, but the reduction won't be anywhere near 99.9%.
  - chosen1111 6y ago
    thank you for the feedback. id love to see the this be open sourced. surely someone is looking to recreate this. is anyone doing so to your knowledge? current barrier as far as i understand is that it costs an estimated 4.6MM USD in compute to recreate.
    - Voloskaya 6y ago
      > is anyone doing so to your knowledge? current barrier as far as i understand is that it costs an estimated 4.6MM USD in compute to recreate. That I know of, no one is looking into doing this in the open source community no. I think it's simply too hard and costly. As I said in another comment, even if an open source org. managed to raise 4.6M$, they still wouldn't be able to do it as the infrastructure needed to train a model of that size is not available publicly on any cloud. The cluster Microsoft built for OpenAI is about 285k CPUs and 10k V100 GPUs, this is absolutly huge. From GPT-3's paper we know the training required 3640 PFLOPs/days of compute. A single V100 GPU is about 14 TFLOPs in half precision, so if you had 75 of those GPUs (which is already massive by open source standard, that's about 5 DGX-2 nodes), it would take about 10 years of training (by that same calculation, it still took about a month on their hypercluster if they used all the GPUs). And that is probably an optimistic estimate since I am using Nvidia's marketing slide to evaluate a V100 FLOPs. Even if somehow an open source organisation was able to get their hand on a cluster half the size of the one OpenAI used, by the time they figure out all the data scraping/cleaning part, found the correct hyper parameters by experimenting on smaller models, and developed the many other things and tricks you need to do to train those models (optimizers, correct checkpointing/resuming strategy etc.) I believe one of OpenAI/Microsoft/Google/Nvidia would already have (or be very close to) announcing the next bigger model, and then it's back to square 1. I think it's just too big a project (with too little usefuleness) to justify undertaking if you are not a massive organization. The closest open source org that could pull this off that I can think of would be HuggingFace. They have the skill and descend funding, and they have the incentive (I believe they also want to enter that NLP API space), but they don't have access to that kind of compute infrastructure (that I know of). They also have been open source so far, but nothing guarantees that would share the checkpoints of such a model should they train it.
derefr 6y ago
Possibly a hot take: I would say that OpenAI is still “open”, in about the same sense that Nginx or Redis is “open”. They seem to be pursuing an open-core model, where the architecture and skeleton of the project is open, but the extra stuff that lots of corporate money was dedicated to building on top is closed. Remember, GPT-3 is just GPT-2 with (a lot) more training data—training data that cost a lot of money to acquire, and therefore training data that (unlike GPT-2’s training set) has corporate interests invested in its creation. GPT-2 is still the “core” of GPT-3’s architecture; GPT-3 can be seen as a commercial “extension” on top of the GPT-2 core. If you want to understand “how GPT-3 works”, you can just study GPT-2. Nothing is different architecturally between them. It is a shame that GPT-3 isn’t open for study, because it has emergent capabilities (meta-learning) that GPT-2 didn’t express, and which are worthy of study. But they couldn’t have known it would do that when they set off to create GPT-3; and they did know they needed a bunch of money for compute + training-data scraping, so they likely made agreements (handshake or contractual) to license this beefed-up “extension” to GPT-2 in advance of knowing what it could do; and in fact as a prerequisite to getting that funding to build it. I would suspect that OpenAI’s goal with GPT-3 was mostly to observe for themselves what a much-better-trained GPT-2 would be capable of; to study those observed capabilities (like meta-learning), and to try to figure out if they can replicate them using only tweaks to GPT-2’s architecture, without needing the big load of commercially-funded training data. (Sort of like seeing what optimal solutions a brute-force algorithm can find, and then trying to come up with an efficient algorithm to emit those same solutions.) If they can come up with something like that, then there’s likely a GPT-4 brewing right now, which wouldn’t need all that compute and training data to create; and therefore wouldn’t be beholden to corporate interests; and therefore would be open for study.
- jVinc 6y ago
  The "Open" in OpenAI means that they want their customer to open their pockets, it has nothing to do with Open Source, open core or anything like that. The cultural war of "but they deserve to make money" is misguided. You are free to make money, just don't call your projects open source if for one reason or another you do not want to open up the source of your project. And don't call it open core if you don't want to open up the core of your project. GPT-3 might just be "GPT-2 plus something more", but a proprietary code based forked off of Linux would also just be "Linux plus something more", that doesn't make it open source or open core, because neither the core nor the source is open. That said, I am not against the OpenAI project going forwards with early releases of proprietary offerings, it seems like a fine way to finance their work, and they can still support the open source community. But criticism of the choice is warranted if they keep trying to brand themselves as open.
mellosouls 6y ago
Given the frequency, HN needs an automatic OpenAI submission checker that changes all titles to read 'ClosedAI' instead. How far the original initiative has fallen; how smug it now looks.
- amelius 6y ago
  Perhaps someone can write a bookmarklet or browser plugin with the same effect.
propogandist 6y ago
It's part of their 2019 agreement for taking $1B in funding >Multiyear partnership ... and an investment of $1 billion from Microsoft... > Microsoft and OpenAI will jointly build new Azure AI supercomputing technologies > OpenAI will port its services to run on Microsoft Azure, which it will use to create new AI technologies and deliver on the promise of artificial general intelligence >> Microsoft will become OpenAI’s preferred partner for commercializing new AI technologies https://news.microsoft.com/2019/07/22/openai-forms-exclusive-computing-partnership-with-microsoft-to-build-new-azure-ai-supercomputing-technologies/ https://news.microsoft.com/2019/07/22/openai-forms-exclusive...
- jonplackett 6y ago
  Financially it kinda makes sense. But isn’t this contrary to the main reason for OpenAI’s existence - to be... open?
deleted 6y ago
[deleted]
freakynit 6y ago
OpenAI: This technology can be misused hence we'll not release it to the public. Also OpenAI: Exclusive access to Microsoft which eventually will release it to public but after paying to OpenAI. Elon is noone more than a typical corporate who slips away from his ethical commitments at the first opportunity of making money.
- joshuaissac 6y ago
  > Elon is noone more than a typical corporate who slips away from his ethical commitments at the first opportunity of making money. What does it have to do with Elon Musk? The article does not mention him as the cause of OpenAI's actions, and he himself has said that he does not have control of the company: "I have no control & only very limited insight into OpenAI."[1] 1. https://twitter.com/elonmusk/status/1229546206948462597 https://twitter.com/elonmusk/status/1229546206948462597
  - junon 6y ago
    The full tweet mentions Dario, too. The full tweet text: > I have no control & only very limited insight into OpenAI. Confidence in Dario for safety is not high. Dario is the research director by the way. This tweet says a lot, in my opinion.
  - freakynit 6y ago
    Fair point. Then his name shouldn't either be used to "market" it and gain public rapport. Because the impression it gives to public is him being in control all while there's no truth in it and public falsely believing in OpenAI's commitments assuming Elon is associated with it. My question: Why has Elon not put his departure news on twitter as his own tweet rather than just replying to someone? Here are a few pages on openai's website that still carry his name. 1. https://openai.com/blog/authors/elon/ https://openai.com/blog/authors/elon/ 2. "Additionally, Elon Musk will depart the OpenAI Board but will continue to donate and advise the organization" (https://openai.com/blog/openai-supporters/ https://openai.com/blog/openai-supporters/) 3. https://openai.com/blog/openai-technical-goals/ https://openai.com/blog/openai-technical-goals/ 4. https://openai.com/blog/introducing-openai/ https://openai.com/blog/introducing-openai/
versteegen 6y ago
Who said Microsoft is getting an "exclusive license"? The OpenAI blog post doesn't. It's licensed to Microsoft now, and presumably to other companies in future (to not do so would be leaving money on the table).
- webmaven 6y ago
  > Who said Microsoft is getting an "exclusive license"? The OpenAI blog post doesn't. It's licensed to Microsoft now, and presumably to other companies in future (to not do so would be leaving money on the table). It's right there in the article, including a link to Microsoft's announcement, which says, among other things: "Today, I’m very excited to announce that Microsoft is teaming up with OpenAI to exclusively license GPT-3 [...]" I'm pretty sure that when Microsoft claims they have an exclusive license to some IP, they actually do.
  - versteegen 6y ago
    Thanks. I didn't see Microsoft's announcement.
JDulin 6y ago
OpenAI's need for these types of "closed" deals is driven by the fact their technology is fundamentally dependent on data & compute infrastructure that only a few organizations in the world can afford. It doesn't matter how many white papers they publish making AI "open" if it costs $100Ms to train & deploy. You could do more to make AI "Open" by working on semiconductors to bring design, test, and fab costs down by orders of magnitude or working on modeling which drastically reduces training & data requirements.
slaymaker1907 6y ago
I think the even bigger issue than something like GPT-3 is how inaccessible a lot of these ML techniques are without huge datasets that are only available to huge companies like FAANG and social media companies. While Twitter is definitely not the best site for a lot of research, it ends up getting used for tons of research because it is one of the only companies that allows others to access even a sample of its data without going through lots of hoops. I think opening up the models is a good start, but I think it is even more important that we allow responsible use of these huge datasets. I also think that ensuring responsibility probably needs to be done by governments rather than companies since allowing companies to apply a lot of restrictions to data access leads to them cutting access off to everyone. If we need to restrict access to certain data because it might be abused, either the collection of that data needs to be banned for all companies or maybe we need to place restrictions on how that data is processed/used rather than restricting its access to a few people. Disclosure: I work as a SE at Microsoft.
- amelius 6y ago
  Yes, I think the people should be in charge of their data. And if a company grows because of user data, then that company should be owned by the people who fed it that data. In the old days we would use telephony to talk to each other, and even the thought of a company listening in on the conversation was abhorrent. But nowadays it seems like it's business as usual. What went wrong?
- logicslave 6y ago
  Actually its not the dataset that matters, its the access to unbounded compute. GPT-3 is trained on data scraped from the web...
cvaidya1986 6y ago
These models will be a commodity with major cloud providers providing near similar performance in the next few years. Will getting early access provide an advantage to startups ? Probably but might fall into the SISP category.
marmaduke 6y ago
The GPT-3 paper clearly states that they used a MS GPU cluster, with estimated cost elsewhere in what, millions? Why is it a surprise then, that they get exclusive access? I think the real story is something like, the rockstar ML team at OpenAi sells its projects to the highest bidder in terms of hardware access, and Google and Amazon just missed the boat on it.
oefrha 6y ago
I just whipped up a Chrome/Firefox extension to replace all instances of "OpenAI" with "ClosedAI" (or anything else you like) on the web, in the spirit of the legendary Cloud to Butt. Help yourself. https://github.com/clopen/ClosedAI https://github.com/clopen/ClosedAI
- ClosedAI 6y ago
  :thumbs_up:
- anaganisk 6y ago
  Bullshit JS, add OPENI to the dictionary
  - still_grokking 6y ago
    Why the down votes? Does it contain "trigger words"? ;-) I think even the down voters would have liked it if they would have looked it up! See yourself: https://mourner.github.io/bullshit.js/ https://mourner.github.io/bullshit.js/
he11ow 6y ago
Contrarian Opinion: It doesn't matter one bit. Take a walk down any science museum, and you see a consistent pattern in technology development: people will push an existing technology to its absolute extreme before they come up with the next clever step. That's GPT-3. It's no holy grail. I find work of the kind that Hugging Face are doing much more interesting, because they are trying to move forward. Their approach is trying to overcome two stumbling blocks for which GPT-3 has no answer: One, that useful language is domain-specific (in the sense that, beyond the written words, it presupposes a familiarity with a world of context). And two, that the way forward is not going big, it's going small. In practically every aspect of historical technology development, the vector for improvement was towards being able to do more with less. These are not the only problem GPT has. Other potholes include its blindness to text structure. This includes logic, and also why it has this rambling incoherent style. So to me, GPT-3 never really crosses over from an academic curio to something that allows us to do something we've not done before. Can it somehow be used to sell EVEN MORE ads? Possibly. Hoorah. But it's not really the milestone the Open AI would like you to believe. (However, if anyone can package Elon Musk's capacity to spin...)
mirekrusin 6y ago
I think by “democratise ai “ they meant “commercialise ai” they should fix that typo in their mission statement.
photonios 6y ago
In the blog post by OpenAI it doesn't mention it's an exclusive deal: https://openai.com/blog/openai-licenses-gpt-3-technology-to-microsoft/ https://openai.com/blog/openai-licenses-gpt-3-technology-to-...
npv789 6y ago
MS AI !

172 points by ColinWright 6y ago

Read this: https://openai.com/charter/ https://openai.com/charter/ Then read this: https://openai.com/blog/openai-licenses-gpt-3-technology-to-microsoft/ https://openai.com/blog/openai-licenses-gpt-3-technology-to-... (the very first dot point) Dear OpenAI, please provide more details of the arrangement with Microsoft. The lack of transparency here shows a clear departure from your charter.

amelius 6y ago
Can someone who has done work for OpenAI sue them for not sticking with their charter?
jonathathan 6y ago
The charter only constrains their use of AGI, which they define at the top in a way that excludes all of their current and near-future work.
- mkolodny 6y ago
  "We commit to use any influence we obtain over AGI’s deployment to ensure it is used for the benefit of all, and to avoid enabling uses of AI or AGI that harm humanity or unduly concentrate power." That sounds to me like they committed to avoid using any AI to concentrate power.
  - still_grokking 6y ago
    But it is written: "unduly" concentrate. The whole sentence is just yada-yada anyway. All "commitments" are constrained by some undefined or impossible exception. Defining "unduly" would be even considerably easy compared to finding a definition for what could be meant by "harm humanity" or be "for the benefit of all". The last constrain is nonsense in itself actually, or does "all" except Microsoft who would not benefit if they wouldn't got an exclusive contract? Not to mention all the people who will lose their jobs as those jobs will be replaced by AI eventually… They didn't say "all equally" which would make at least some little bit sense.
cscurmudgeon 6y ago
This is a very vexing situation. I applied for access months ago. I am a scientist and academic with dozens of articles on topics related to GPT (and especially survey articles and a few articles reasonably critical of some NLP/ML/DL frameworks). I am not sure I can reveal more without doxing myself. I haven't heard anything from them yet. They are free to not release GPT-3 but not free to mislead the open community with this open and holier-than-thou nonsense.

yowlingcat 6y ago

I really can't stand the bait and switch branding of OpenAI. They are anything but open, and there is no reason to believe their tall tale that the only reason they don't release the model publicly is "ethical" reasons -- it's for monetary reasons and it's gatekeeping. Not only that, but they are playing a dangerous game strategically. Companies like Hugging Face actually release valuable parts of the open source NLP ecosystem which set new records for SOTA and which the rest of the industry can use. For all the practitioners here who probably know more about this than I do, help me understand: how much of OpenAI is smoke and mirrors? Is it just me, or is GPT-3 just GPT-2 but with way more training data? Are there fundamental conceptual differences with how the scale of that data is handled?

junon 6y ago
Pretty much share this sentiment. I read their FAQ page and it left a really bad taste in my mouth. A further question that I'd like to add - are there any _truly_ open source initiatives that can remotely compete with GPT-3? What's stopping the OSS community (specifically, not just "funding")? Is it lack of compute power? Lack of data?
- ForHackernews 6y ago
  I think it's just money. I'm not an expert, but my understanding is there's nothing fundamentally new or revolutionary about GPT-3. It's not a new AI paradigm, it's just taking existing techniques and turning them up to 11. If you throw absolutely enormous amounts of training data and compute resources at a bigass neural network, you get surprisingly good results. Anyone else with a few tens of millions of dollars could do similarly.
- est31 6y ago
  For GPT-3, I think they mainly scraped stuff off the internet. Anyone with enough time and machines can replicate that, but it is a non zero effort. GPT is an exception here I think. It's harder to build open source MRI diagnosis tools for example when nobody gives you open source MRI scans with diagnosis data. It's more a lack of compute power problem plus a lack of people problem. The compute cost alone was estimated to be 4.6 million USD. As for the people, folks who can push SOTA are extremely sought after. So the individuals involved also pay a large opportunity cost by working on things for free or comparatively small open source pay. Also, I think many open source organizations/communities haven't recognized the value of open source ML models yet and don't invest in them as much as in parametric models. I think this will come with time. Last, anyone with a computer can write code, while few people have access to the hardware needed.

nitinreddy88 6y ago

Old news and already discussed https://news.ycombinator.com/item?id=24558329 https://news.ycombinator.com/item?id=24558329

ackbar03 6y ago

Well Microsoft did give them some sort of discount or something for their azure servers for training. I think it was part of the agreement or something

webmaven 6y ago
> Well Microsoft did give them some sort of discount or something for their azure servers for training. I think it was part of the agreement or something I think about half of the money Microsoft invested in OpenAI has been cash, the rest was in the form of Azure credits.
- webmaven 6y ago
  I realize it isn't obvious why this probably precludes a discount. For any given deal, it is much simpler for all parties to adjust the amount of credits being given than to negotiate a discount.

jgalt212 6y ago

open != exclusive

TaupeRanger 6y ago

Thus changing its name to ExclusiveAI? No? Still the air of moral superiority?

DaiPlusPlus 6y ago
When Progress Inc. changed the name of their awful-yet-surprisingly-popular closed-source (even [anti-open-source](https://www.progress.com/tutorials/odbc/open-source-database https://www.progress.com/tutorials/odbc/open-source-database) ) 4GL from ProgressDB to "OpenEdge" it was a reminder to me that there is absolutely nothing to a name. "A rose by any other name would smell as sweet" - similarly: "Open[BLANK] by any other name would be just as much marketing bollocks".
- PeterStuer 6y ago
  What does this have to do with the discussion about OpenAI changing not just strategy but what feels to many as changing their core values that they always promoted as being the central reason for them existing? "But what about ..." is irrelevant https://en.wikipedia.org/wiki/Whataboutism https://en.wikipedia.org/wiki/Whataboutism
  - DaiPlusPlus 6y ago
    > What does this have to do with the discussion about OpenAI changing not just strategy Nothing to do with that. Nor is this a case of whataboutism. I am not justifying one company's actions based on another unrelated company's poor business decisioins. I am stating that given this deal with Microsoft, the "Open" in OpenAI's name is now meaningless and/or dishonest. I used the example of Progress/OpenEdge as another example of a company putting "open" right into their name but failing to live up to their name - implying that Progress' name-change (and also: OpenAI's initial name) was a cynical marketing ploy to make people believe they're the _good guys_ in all this, especially when the AI/ML field has many detractors from the altruistic camps, especially after Deep-Fakes, Palantir, Cambridge Analytica, and Facebook have all left a bad taste in everyone's' mouth.

simonkafan 6y ago

I still don't see or have read about a useful use case for GPT-3. Maybe for procedurally generated chitchat between NPC in video games or an advanced Lorem ipsum generator but that's about it.

bananaface 6y ago
Auto-formatting punctuation - or syntax. You can show it a few examples and it will insert punctuation into raw words coming in from speech recognition, or convert badly formatted Python into well-formatted Python. It can correct syntax errors in C++ (!). More generally, it can convert text from a simple, unstructured form into more complex, more structured form, with just a few example transforms. It's astonishingly good at it, but I haven't seen much on the public net about it. I don't know why people aren't exploring it more tbh.
- tensility 6y ago
  Now that it is a captive technology, the question becomes why should anyone want to explore it other than Microsoft.
  - bananaface 6y ago
    It's not captive. It's quite replicable (and impossible to monopolise), it's just expensive.
TaupeRanger 6y ago
Yes. People were literally claiming to be able to write code using English language descriptions. This guy on Twitter generated a ton of hype with these cherry picked examples...even from people who should know better like Eliezer Yudkowsky. https://twitter.com/sharifshameem/status/1284095222939451393?s=19 https://twitter.com/sharifshameem/status/1284095222939451393...
- avoidboringppl 6y ago
  Also interesting to note that the hype came a while after the release of GPT-3 (the tweet was about a month after), and that many of the examples could have possibly been generated from GPT-2 (which didn't get as much hype). Put another way, you could show ppl GPT-2 examples from a year ago and claim they were from GPT-3, and I don't think many would know the difference. I do think GPT-3 has shown improvement, and it's a step forward, but it probably tells more about how humans interpret rather than how AI might work. Wrote about it more in the link below: https://avoidboringpeople.substack.com/p/doctor-gpt-3 https://avoidboringpeople.substack.com/p/doctor-gpt-3

chosen1111 6y ago

Lets just build another and be done with it. It’s expensive to make with current parameters. Can anyone weigh in on the following as a means of lowering compute cost. https://www.infoq.com/news/2020/10/training-exceeds-gpt3/ https://www.infoq.com/news/2020/10/training-exceeds-gpt3/

Voloskaya 6y ago
This thing is pure clickbait. It's better than GPT-3 on language representation tasks where GPT-3 was not very good to start with. GPT-3 is a language generation model, and on language generation tasks, GPT-3 is absolutely better than anything else right now. In PET's author own words: "GPT-3 certainly is much better than our approach at generating long sequences of text (e.g., summarization or machine translation)" There are things that can be done to have something similar to GPT-3 in performance at a lower cost, but the reduction won't be anywhere near 99.9%.
- chosen1111 6y ago
  thank you for the feedback. id love to see the this be open sourced. surely someone is looking to recreate this. is anyone doing so to your knowledge? current barrier as far as i understand is that it costs an estimated 4.6MM USD in compute to recreate.
  - Voloskaya 6y ago
    > is anyone doing so to your knowledge? current barrier as far as i understand is that it costs an estimated 4.6MM USD in compute to recreate. That I know of, no one is looking into doing this in the open source community no. I think it's simply too hard and costly. As I said in another comment, even if an open source org. managed to raise 4.6M$, they still wouldn't be able to do it as the infrastructure needed to train a model of that size is not available publicly on any cloud. The cluster Microsoft built for OpenAI is about 285k CPUs and 10k V100 GPUs, this is absolutly huge. From GPT-3's paper we know the training required 3640 PFLOPs/days of compute. A single V100 GPU is about 14 TFLOPs in half precision, so if you had 75 of those GPUs (which is already massive by open source standard, that's about 5 DGX-2 nodes), it would take about 10 years of training (by that same calculation, it still took about a month on their hypercluster if they used all the GPUs). And that is probably an optimistic estimate since I am using Nvidia's marketing slide to evaluate a V100 FLOPs. Even if somehow an open source organisation was able to get their hand on a cluster half the size of the one OpenAI used, by the time they figure out all the data scraping/cleaning part, found the correct hyper parameters by experimenting on smaller models, and developed the many other things and tricks you need to do to train those models (optimizers, correct checkpointing/resuming strategy etc.) I believe one of OpenAI/Microsoft/Google/Nvidia would already have (or be very close to) announcing the next bigger model, and then it's back to square 1. I think it's just too big a project (with too little usefuleness) to justify undertaking if you are not a massive organization. The closest open source org that could pull this off that I can think of would be HuggingFace. They have the skill and descend funding, and they have the incentive (I believe they also want to enter that NLP API space), but they don't have access to that kind of compute infrastructure (that I know of). They also have been open source so far, but nothing guarantees that would share the checkpoints of such a model should they train it.

derefr 6y ago

Possibly a hot take: I would say that OpenAI is still “open”, in about the same sense that Nginx or Redis is “open”. They seem to be pursuing an open-core model, where the architecture and skeleton of the project is open, but the extra stuff that lots of corporate money was dedicated to building on top is closed. Remember, GPT-3 is just GPT-2 with (a lot) more training data—training data that cost a lot of money to acquire, and therefore training data that (unlike GPT-2’s training set) has corporate interests invested in its creation. GPT-2 is still the “core” of GPT-3’s architecture; GPT-3 can be seen as a commercial “extension” on top of the GPT-2 core. If you want to understand “how GPT-3 works”, you can just study GPT-2. Nothing is different architecturally between them. It is a shame that GPT-3 isn’t open for study, because it has emergent capabilities (meta-learning) that GPT-2 didn’t express, and which are worthy of study. But they couldn’t have known it would do that when they set off to create GPT-3; and they did know they needed a bunch of money for compute + training-data scraping, so they likely made agreements (handshake or contractual) to license this beefed-up “extension” to GPT-2 in advance of knowing what it could do; and in fact as a prerequisite to getting that funding to build it. I would suspect that OpenAI’s goal with GPT-3 was mostly to observe for themselves what a much-better-trained GPT-2 would be capable of; to study those observed capabilities (like meta-learning), and to try to figure out if they can replicate them using only tweaks to GPT-2’s architecture, without needing the big load of commercially-funded training data. (Sort of like seeing what optimal solutions a brute-force algorithm can find, and then trying to come up with an efficient algorithm to emit those same solutions.) If they can come up with something like that, then there’s likely a GPT-4 brewing right now, which wouldn’t need all that compute and training data to create; and therefore wouldn’t be beholden to corporate interests; and therefore would be open for study.

jVinc 6y ago
The "Open" in OpenAI means that they want their customer to open their pockets, it has nothing to do with Open Source, open core or anything like that. The cultural war of "but they deserve to make money" is misguided. You are free to make money, just don't call your projects open source if for one reason or another you do not want to open up the source of your project. And don't call it open core if you don't want to open up the core of your project. GPT-3 might just be "GPT-2 plus something more", but a proprietary code based forked off of Linux would also just be "Linux plus something more", that doesn't make it open source or open core, because neither the core nor the source is open. That said, I am not against the OpenAI project going forwards with early releases of proprietary offerings, it seems like a fine way to finance their work, and they can still support the open source community. But criticism of the choice is warranted if they keep trying to brand themselves as open.

mellosouls 6y ago

Given the frequency, HN needs an automatic OpenAI submission checker that changes all titles to read 'ClosedAI' instead. How far the original initiative has fallen; how smug it now looks.

amelius 6y ago
Perhaps someone can write a bookmarklet or browser plugin with the same effect.

propogandist 6y ago

It's part of their 2019 agreement for taking $1B in funding >Multiyear partnership ... and an investment of $1 billion from Microsoft... > Microsoft and OpenAI will jointly build new Azure AI supercomputing technologies > OpenAI will port its services to run on Microsoft Azure, which it will use to create new AI technologies and deliver on the promise of artificial general intelligence >> Microsoft will become OpenAI’s preferred partner for commercializing new AI technologies https://news.microsoft.com/2019/07/22/openai-forms-exclusive-computing-partnership-with-microsoft-to-build-new-azure-ai-supercomputing-technologies/ https://news.microsoft.com/2019/07/22/openai-forms-exclusive...

jonplackett 6y ago
Financially it kinda makes sense. But isn’t this contrary to the main reason for OpenAI’s existence - to be... open?

deleted 6y ago

[deleted]

freakynit 6y ago

OpenAI: This technology can be misused hence we'll not release it to the public. Also OpenAI: Exclusive access to Microsoft which eventually will release it to public but after paying to OpenAI. Elon is noone more than a typical corporate who slips away from his ethical commitments at the first opportunity of making money.

joshuaissac 6y ago
> Elon is noone more than a typical corporate who slips away from his ethical commitments at the first opportunity of making money. What does it have to do with Elon Musk? The article does not mention him as the cause of OpenAI's actions, and he himself has said that he does not have control of the company: "I have no control & only very limited insight into OpenAI."[1] 1. https://twitter.com/elonmusk/status/1229546206948462597 https://twitter.com/elonmusk/status/1229546206948462597
- junon 6y ago
  The full tweet mentions Dario, too. The full tweet text: > I have no control & only very limited insight into OpenAI. Confidence in Dario for safety is not high. Dario is the research director by the way. This tweet says a lot, in my opinion.
- freakynit 6y ago
  Fair point. Then his name shouldn't either be used to "market" it and gain public rapport. Because the impression it gives to public is him being in control all while there's no truth in it and public falsely believing in OpenAI's commitments assuming Elon is associated with it. My question: Why has Elon not put his departure news on twitter as his own tweet rather than just replying to someone? Here are a few pages on openai's website that still carry his name. 1. https://openai.com/blog/authors/elon/ https://openai.com/blog/authors/elon/ 2. "Additionally, Elon Musk will depart the OpenAI Board but will continue to donate and advise the organization" (https://openai.com/blog/openai-supporters/ https://openai.com/blog/openai-supporters/) 3. https://openai.com/blog/openai-technical-goals/ https://openai.com/blog/openai-technical-goals/ 4. https://openai.com/blog/introducing-openai/ https://openai.com/blog/introducing-openai/

versteegen 6y ago

Who said Microsoft is getting an "exclusive license"? The OpenAI blog post doesn't. It's licensed to Microsoft now, and presumably to other companies in future (to not do so would be leaving money on the table).

webmaven 6y ago
> Who said Microsoft is getting an "exclusive license"? The OpenAI blog post doesn't. It's licensed to Microsoft now, and presumably to other companies in future (to not do so would be leaving money on the table). It's right there in the article, including a link to Microsoft's announcement, which says, among other things: "Today, I’m very excited to announce that Microsoft is teaming up with OpenAI to exclusively license GPT-3 [...]" I'm pretty sure that when Microsoft claims they have an exclusive license to some IP, they actually do.
- versteegen 6y ago
  Thanks. I didn't see Microsoft's announcement.

JDulin 6y ago

OpenAI's need for these types of "closed" deals is driven by the fact their technology is fundamentally dependent on data & compute infrastructure that only a few organizations in the world can afford. It doesn't matter how many white papers they publish making AI "open" if it costs $100Ms to train & deploy. You could do more to make AI "Open" by working on semiconductors to bring design, test, and fab costs down by orders of magnitude or working on modeling which drastically reduces training & data requirements.

slaymaker1907 6y ago

I think the even bigger issue than something like GPT-3 is how inaccessible a lot of these ML techniques are without huge datasets that are only available to huge companies like FAANG and social media companies. While Twitter is definitely not the best site for a lot of research, it ends up getting used for tons of research because it is one of the only companies that allows others to access even a sample of its data without going through lots of hoops. I think opening up the models is a good start, but I think it is even more important that we allow responsible use of these huge datasets. I also think that ensuring responsibility probably needs to be done by governments rather than companies since allowing companies to apply a lot of restrictions to data access leads to them cutting access off to everyone. If we need to restrict access to certain data because it might be abused, either the collection of that data needs to be banned for all companies or maybe we need to place restrictions on how that data is processed/used rather than restricting its access to a few people. Disclosure: I work as a SE at Microsoft.

amelius 6y ago
Yes, I think the people should be in charge of their data. And if a company grows because of user data, then that company should be owned by the people who fed it that data. In the old days we would use telephony to talk to each other, and even the thought of a company listening in on the conversation was abhorrent. But nowadays it seems like it's business as usual. What went wrong?
logicslave 6y ago
Actually its not the dataset that matters, its the access to unbounded compute. GPT-3 is trained on data scraped from the web...

cvaidya1986 6y ago

These models will be a commodity with major cloud providers providing near similar performance in the next few years. Will getting early access provide an advantage to startups ? Probably but might fall into the SISP category.

marmaduke 6y ago

The GPT-3 paper clearly states that they used a MS GPU cluster, with estimated cost elsewhere in what, millions? Why is it a surprise then, that they get exclusive access? I think the real story is something like, the rockstar ML team at OpenAi sells its projects to the highest bidder in terms of hardware access, and Google and Amazon just missed the boat on it.

oefrha 6y ago

I just whipped up a Chrome/Firefox extension to replace all instances of "OpenAI" with "ClosedAI" (or anything else you like) on the web, in the spirit of the legendary Cloud to Butt. Help yourself. https://github.com/clopen/ClosedAI https://github.com/clopen/ClosedAI

ClosedAI 6y ago
:thumbs_up:
anaganisk 6y ago
Bullshit JS, add OPENI to the dictionary
- still_grokking 6y ago
  Why the down votes? Does it contain "trigger words"? ;-) I think even the down voters would have liked it if they would have looked it up! See yourself: https://mourner.github.io/bullshit.js/ https://mourner.github.io/bullshit.js/

he11ow 6y ago

Contrarian Opinion: It doesn't matter one bit. Take a walk down any science museum, and you see a consistent pattern in technology development: people will push an existing technology to its absolute extreme before they come up with the next clever step. That's GPT-3. It's no holy grail. I find work of the kind that Hugging Face are doing much more interesting, because they are trying to move forward. Their approach is trying to overcome two stumbling blocks for which GPT-3 has no answer: One, that useful language is domain-specific (in the sense that, beyond the written words, it presupposes a familiarity with a world of context). And two, that the way forward is not going big, it's going small. In practically every aspect of historical technology development, the vector for improvement was towards being able to do more with less. These are not the only problem GPT has. Other potholes include its blindness to text structure. This includes logic, and also why it has this rambling incoherent style. So to me, GPT-3 never really crosses over from an academic curio to something that allows us to do something we've not done before. Can it somehow be used to sell EVEN MORE ads? Possibly. Hoorah. But it's not really the milestone the Open AI would like you to believe. (However, if anyone can package Elon Musk's capacity to spin...)

mirekrusin 6y ago

I think by “democratise ai “ they meant “commercialise ai” they should fix that typo in their mission statement.

photonios 6y ago

In the blog post by OpenAI it doesn't mention it's an exclusive deal: https://openai.com/blog/openai-licenses-gpt-3-technology-to-microsoft/ https://openai.com/blog/openai-licenses-gpt-3-technology-to-...

npv789 6y ago

MS AI !