11 ms·

AI Democratization in the Era of GPT-3

86 points by jonbaer 6y ago

data_ders 6y ago
I do agree that it isn’t “democratization” in the sense of “having access” and that it’s a buzz word. But with Azure Cognitive Services and Azure AutoML, I do think MSFT has shown itself capable of point 3: enabling its customers “to use the algorithms and models, potentially without requiring advanced mathematical and computing science skills”.
andreyk 6y ago
Curious as to what people think wrt the final question " Do we trust [OpenAI] to take on this role [of deciding who can have access to newer SOTA models and who can't]? And if not, how can academics and practitioners fight for the continued democratization of AI, as some of its most important techniques become as hard to replicate as GPT-3?" It seems like for super gigantic models, good infra to train on cloud deployment (for which many labs can have credits / grants) would be the first priority, followed perhaps by good compression/pruning tech so after training inference can be done on one/fewer GPUs. Any other things?
- visarga 6y ago
  Maybe the authors of the text should also have a stake in GPT-3. After all, OpenAI didn't write the corpus. Google benefits from the web, and Facebook from the real-life social networks and their activities that it replicates online (messages, meetings, news, etc). We are all the source of the training data. Why should we be at the whims of these derivative product companies?
  - andreyk 6y ago
    Fun fact, OpenAI did not collect the data itself - they mainly used data from Common Crawl (in addition to a couple other datasets), which is compiled by a non profit that shares the dataset for free. So perhaps the license of such datasets can encourage free sharing of research outcomes. https://commoncrawl.org/ https://commoncrawl.org/
    - searchableguy 6y ago
      Shouldn't I own part of gpt3 because it's trained on my data and likely spitting out what I have commented somewhere on the internet? I think ml models should be public unless the data itself isn't.
      - searchableguy 6y ago
        Curious about the downvotes. If you remove compute and code, what remains is data that isn't owned by a single entity in the case of gpt3. You can sell both compute and code which is what you own but should you be able to sell data? You can't translate a commerical book without paying the copyright holder. You own the copyright for the translation but everything else still remains that of the original author. Why wouldn't this apply in case of a machine learning model trained on my images, messages, intents, likeness, etc?
        isx726552 6y ago
        But you don’t understand, OpenAI has hype and “rockstars” and big names attached, so those rules don’t apply. Your data? It’s their data now. (And Microsoft’s.)
        PeterisP 6y ago
        The current legal understanding is that according to the current copyright laws the ML models trained on some data are not considered derivative works of that data, so if the data was obtained legally and without other restrictions (e.g. if the researchers sign a contract with the data owner that gives them access to data but agree to certain conditions) then the authors of the data do not have any claim on some rights to the resulting model. One aspect of this is that historically statistical models calculated from large volumes of text (which is a notion that predates computers, e.g. frequency dictionaries and the whole [sub]field of quantitative corpus linguistics) have been considered facts about that corpus of text and thus not copyrightable at all or (depending on jurisdiction) entitled to different set of protections/limitations assigned to compilations of facts, which give some rights to the people who compiled the facts but no rights to the source of these facts (since facts as such can't be copyrighted). This also applies to many forms of analysis of audiovisual data, where the copyrights of the source works do not transfer to the results of the statistical or qualitative analysis and can't limit their creation, distribution or sale. The appropriate analogy to a commercial book or movie is not a translation, but some analysis of it - e.g. a thorough literary review and critique of some book or movie is a separate work with its own copyright, and the original author has no claim on it despite the fact that is (obviously) based on the contents of the work and describes it in great detail. Including verbatim fragments of the work is limited (fair use allows some inclusions but not all), but all the other details are not. Also, the whole notion of copyrightability of ML model weight files is interesting and IMHO not settled. You could argue that there is some creative expression in forming the model (which would support it being copyrightable) or you could argue that it's a mechanistic result of the application of some algorithm and settings (which have the creative part, and are copyrightable on their own), and so the output can't be copyrightable, no matter how much work (human or machine), time and cost it took - at least in USA copyright law doctrine (e.g. Feist Publications v. Rural Telephone Service) is that mere "sweat of the brow" (no matter how much) does not entitle a work to copyright protection; it requires application of human creativity to create an original work, and automated processes can't satisfy that requirement. So from the perspective of GPT-3 authors it might be prudent to distribute the model only to people who agree to specific, explicit contractual/licensing restrictions on redistribution and usage of the model instead of simply relying on the default application of copyright laws. And crucially, if some output is not copyrightable, it can't be considered a derived work to which the authors of a source work could have some claim.
- whichquestion 6y ago
  Would we be able to democratize these large ML models using something like SETI@Home or similar? Would more research in distributing models across large networks be useful? Has research on this topic been done?
ve55 6y ago
OpenAI's goal is definitely not to give everyone unlimited/equal access to powerful tools like GPT-3. We've had countless jokes about the name being 'OpenAI', and perhaps it's true that it's not the best name (along with 'democratizing' AI), but I'm not sure the author is suggesting a solution here rather than just venting that things seem kind of unfair, and no one outside of OpenAI really has much control or information available such as what he asks about. But I personally find the complaints to be understandable, especially as someone that didn't get a response for my requests for GPT-3 beta access, it felt pretty bad to watch everyone else have fun building cool things with the world's best text AI while I sat there and couldn't do anything, even if I was willing to pay for access. Hopefully there will be other relevant players here besides just OpenAI sooner or later.
- langitbiru 6y ago
  "other relevant players" -> Google could create one.
- ma2rten 6y ago
  The article is hinting at this but I also think many people who complain that OpenAI didn't release the model don't understand how big this model actually is. Even if they had access to the parameters they couldn't do much with it. Assuming you used single precision the model is 350 gigabytes (175 billion * 2 bytes). For fast inference the model needs to be in GPU memory. Most GPUs have 16GB of memory, so you would need 22 GPUs just to hold the model in memory and that doesn't even include the memory for activations. If you wanted to do fine tuning, you would need 3x as much memory for gradients and momentum.
  - tarboreus 6y ago
    If it was open, there would be other services offering this, and not just an opaque beta and now a single expensive service.
  - wslh 6y ago
    Tell cryptocurrency miners that this is a big model to compute... the size of this problem seems very tiny. If there are millions of ASIC, GPU, etc devices mining cryptocurrencies it is fair to speculate that democratizing AI has a special room in this model.
angel_j 6y ago
Forget AI democritization, GPT-3 is AI demoguerization. GPT-3 is singular; it is one model, one dataset, one training. Yet it will be the only one that will exist for quite some time (or by far the most available), and now it will underwrite productization and malfeasance, a la mode pay to play. For example, I recently read a paper supposedly written by a Chinese dissident virologist, which report was disseminated by a group with questionable membership. Most of the jargon in the report going over my head, I had to wonder if the otherwise convincing verbiage wasn't the work of GPT-3.
boulos 6y ago
Disclosure: I work on Google Cloud and have worked with the OpenAI folks on large models. This article mixes both “should research be open” and “is this work cheaply reproduced / accessible”: For smaller, open models: > The average person could not recreate models of this size from scratch, but the models can run on a single machine with a single GPU. but about GPT-3: > GPT-3 represents a new circumstance. For the first time, a model is so big it cannot be easily moved to another cloud and certainly does not run on a single computer with a single or small number of GPUs. Instead OpenAI is providing an API so that the model can be run on their cloud. While I’d quibble with “for the first time” (it’s easy to generate mega models! Plenty of distributed mesh tensorflow stuff does that, etc.), I don’t think this is any different than large physics simulations. Is it “wrong” to have some groups push the boundary of what’s possible with supercomputers? I certainly don’t think so. If anything, it shows what’s possible and others can do the valuable work of “miniaturization”. In this specific area, PRADO is a good example relative to BERT. For my historic area of ray tracing research, we did lots of things on an SGI Origin that let us “jump ahead a few years” versus what we could have done on any basic workstation. You could argue that it’s not academically interesting (“you just ran this really big because you had the budget / hardware, whatever”) and reject the paper. [Edit: I consider this kind of work interesting from a systems perspective, not “ML”, but it’s still interesting!] I don’t think it makes sense to suggest that we should hold back progress based on NSF grant funding or least common denominator computing resources. How would you decide what’s acceptable? Is a single A100 affordable? Only a T4? Only a laptop? tl;dr: it’s fine to argue about openness and democratization being hollow marketing words. I’m not sure I would conflate openness with “everyone, everywhere should be able to run any scientific work without expense”.
- robmsmt 6y ago
  how big do the models get? are you able to say?
  - boulos 6y ago
    As big as you want, kind of. The challenge, as in large-scale physics, is how many nodes you can stick together with sufficiently high bandwidth (low latency is less important in the ML space, because there are lots of ops per byte, unlike some CFD Simulations that have very few per update). On our Cloud TPU product page [1], we have a single TPU v3 pod with 32 TB of memory. For the most recent MLperf submission, the TPU folks hooked up four of them [2]. There’s obviously a reduction in scalability from doing so (see weak scaling versus strong scaling terminology), but that’s the interesting co-design question: what kind of models can you usefully train in an “even more distributed” mode? Outside of TPUs though, even our single 16x A100 offering has 640 GB all connected by NVLINK (other providers went with 8, so 320 GB of “system memory”) and there are at least a few in a single rack. So the era of TiB scale models is certainly “semi feasible” and “open to all”. The challenge is that you need to also train these for quite some time. 1000 V100s would cost you at least $2000/hr to rent. Many models are sufficiently complicated (not just large) that you end up training them for days and weeks, even with this much compute. So the numbers add up quickly. But just being “big” doesn’t mean “trained for a month on a supercomputer”. [1] https://cloud.google.com/tpu https://cloud.google.com/tpu [2] https://www.google.com/amp/s/cloudblog.withgoogle.com/products/ai-machine-learning/google-breaks-ai-performance-records-in-mlperf-with-worlds-fastest-training-supercomputer/amp/ https://www.google.com/amp/s/cloudblog.withgoogle.com/produc...
29athrowaway 6y ago
Powerful models are trained using large amounts of data that most people do not have access to. They are also trained powerful infrastructure that most people do not have access to. So, to speak of democratization is interpreting the current state of affairs incorrectly.
- boulos 6y ago
  See my sibling comment, but the dataset that the GPT-2 folks did was “just grab whatever people on reddit upvoted”. Any grad student might have done the same (and in a sense, James Hays and Alyosha Efros did something like that long ago with flickr data for im2gps). While GPT-3 is trained on a massive V100 cluster, you could probably do so with a much smaller one / there exist interesting smaller models. It’s expensive to rent this class of equipment, but it is available. The distinction is that OpenAI made a focused bet. Most research funding and labs spread their bets heavily (e.g., each institution or researcher gets $50k/yr of funding). OpenAI takes a different stance, but it’s not clear that they’re spending even as much as say Google, Facebook or other large institutions. It’s also not obvious that you even have to play the same game to get similar results.
- mirekrusin 6y ago
  They trained it on freely available datasets managed by nonprofits as somebody mentioned. Powerfull infrastructure can be democratized in similar way that SETI@home, cryptocurriencies and other P2P projects do it. In theory this can be done.
skybrian 6y ago
It’s certainly annoying not to have access to the API to try a few things. However, optimization has just begun and it seems like there will probably be competing models 100x smaller in a year or two?
m1rr0rman 6y ago
RussianNationalAnthem.mp3
mensetmanusman 6y ago
I learned some things. GTP-3 is so complex the model requires large cloud computing resources to run. Ergo, it is also very expensive to run. Assumption: Bleeding edge AI will require tens of millions of dollars of computation before new network architectures fall out of state space. After this, the models can be pruned to be ran by mere mortals. If this is true, OpenAI will not be able to move to the next level without partnering with very wealthy institutions a la Microsoft. If this is true, those calling for OpenAI to not monetize intermittent progress are essentially preventing next generation discovery, unless they have alternative monetization ideas to generate 8 figures for research.
- choppaface 6y ago
  There will always be models and research that require proprietary hardware and resources and thus cannot be “democratized.” PageRank required a ton of hardware and an index. Non-industry researchers will always be priced out of something. A key concern here with regards to the ethics-of-AI issue is that last year OpenAI refused to release GPT-2 because it was too ‘dangerous’. This year, GPT-3 is suddenly a revenue-generating Microsoft product. The Gradient article linked is one of the more diplomatic ways of calling BS on OpenAI’s strategy. Financial interests taint research in subtle ways, it has for decades, and OpenAI employees being paid $1m in cash salary need reminders of this fact. https://openai.com/blog/better-language-models/ https://openai.com/blog/better-language-models/
- anoncareer0212 6y ago
  Given the worst-case assumptions of people arguing the cloud is required[1], it's a one-time investment of $60,000, $200,000 in the absolute worst case. [1] https://news.ycombinator.com/item?id=24601264 https://news.ycombinator.com/item?id=24601264
  - exged 6y ago
    That's for model inference. For training OpenAI said they used around 3000 PetaFLOPS / days on the largest GPT-3 model. That translates to about 300 Nvidia A100 GPUs if you want training to finish in a month (any slower and your researchers are not going to be able to make much progress). A system like that would cost at least $5M, probably more like $10M.
deleted 6y ago
[deleted]
cblconfederate 6y ago
I don't see where is the monetization potential of GPT3. Creating more spam? Making poetry in bulk? Being SOTA in language tasks doesnt make it useful per se, as NLP measures are rather abstract anyway. As for its technology, it's huge but unless they are hiding some secret sauce, it's a straigtforwardly transformer-based, which means someone somewhere with cheap electricity is already training it on a dump of the entire web. Where's the moat here? Perhaps in some future version that they 'll keep secret? Well it will be a sad day for humanity if they keep its development in secret, given how much its development has benefited from open science around the world.
chime 6y ago
Not saying I am a fan of everything going on in tech world but it is pretty evident that AI is going to end up happening the West World “Rehoboam” big AI brain way than the Jetsons AI maid way, not purely due to corporate interests but rather due to the inherent cost of just making it work. Unless we can have PB/EB worth of storage locally then it’s going to be the cloud. And a cloud API with 1000 EB storage will always outperform your local 10 PB AI model. The most practical approach that I can see would be to minimize the cost of accessing the model, making it free like author suggests for research, students, non-profits etc. and charging more for commercial usage, basically extending the existing cloud model. One thing that I think should be free for everyone would be testing the model for biases. Basically if every AI API to check bias on any topic was free, then it could be improved by anyone including marginalized groups. If GPT-3 thinks all Indians are either doctors, coders, or gas station owners then I would like to be able to test, verify, and offer a patch without any cost, maybe even a reward. Otherwise GPT-4-5-6 will end up throwing away all Indian sounding last names applying for a construction job.
jackcosgrove 6y ago
The PC was supposed to democratize computing. It did, and then Microsoft found a choke point. The internet was supposed to democratize communication. It did, and then Google found a choke point. It seems like this two steps forward, one step back pattern might be the rule rather than the exception. Even the article defines AI democratization in terms of using models rather than training models, as the costs of training sophisticated models seems beyond the common developer even according to idealists. AI in particular seems to be a centralizing technology at this point, given that a model is often a black box to the user. The amount of data as well requires a massive telemetry apparatus, and designing likely models seems the province of people with PhDs. So yes I'm a bit pessimistic that AI will have a democratizing effect on technology or society, at least in the near future.
- srtjstjsj 6y ago
  > The PC was supposed to democratize computing It was?
  - DSingularity 6y ago
    Yes. Before that it was mainframes and those who had access to $ who had access to computing.
  - nl 6y ago
    Yes. The name gives it away. And Microsoft's mission for many years was "a computer in every home".
    - pixl97 6y ago
      "One ring to rule them all, and in the darkness bind them."
- decasteve 6y ago
  The airplane and the radio brought people closer. It also improved the coordination and delivery of mass destruction. Technology can be a rollercoaster ride of benefit and detriment.
sillysaurusx 6y ago
I’m amazed no one pointed out that OpenAI will be dead within 5 years unless it monetizes GPT-3. Democratizing AI is a fine goal, but it’s secondary to not dying. At least, for OpenAI it is. And that should be obvious.
Fordec 6y ago
I applied for API access months ago and still haven't been able to tinker with it. Until it's immediate sign up and not some insider walled garden waitlist, from a developer perspective, OpenAI and GPT-3 is anything but Open. If I was overly pushed to get access, my best chance of success right now appears to be a search on Github for somebody elses leaked API key in a repo somewhere.
nl 6y ago
This a bad take. In 2 years time there will be a quantized, TensorCore optimised equivalent that will run fine on desktop GPUs. We've seen this over and over again and there is no reason to believe this is different.
alphachloride 6y ago
The models are so unwieldy that had the big companies not handled the training and infrastructure, no one would be able to explore these models. We would have had no democratization. Are these big firms are beholden to open up their work to the public for this utopian ideal of democratization?
bionhoward 6y ago
Exclusive licenses for MSFT to tech invented at GOOG isn’t “democratization”
scoot_718 6y ago
Paying for the API is also bad if you wanted to do literally anything with the raw outputs. Have an idea you want to try out on how to sample the text from the model? Tough. Pay your money and get our output. It's useless.

86 points by jonbaer 6y ago

I do agree that it isn’t “democratization” in the sense of “having access” and that it’s a buzz word. But with Azure Cognitive Services and Azure AutoML, I do think MSFT has shown itself capable of point 3: enabling its customers “to use the algorithms and models, potentially without requiring advanced mathematical and computing science skills”.

andreyk 6y ago

Curious as to what people think wrt the final question " Do we trust [OpenAI] to take on this role [of deciding who can have access to newer SOTA models and who can't]? And if not, how can academics and practitioners fight for the continued democratization of AI, as some of its most important techniques become as hard to replicate as GPT-3?" It seems like for super gigantic models, good infra to train on cloud deployment (for which many labs can have credits / grants) would be the first priority, followed perhaps by good compression/pruning tech so after training inference can be done on one/fewer GPUs. Any other things?

visarga 6y ago
Maybe the authors of the text should also have a stake in GPT-3. After all, OpenAI didn't write the corpus. Google benefits from the web, and Facebook from the real-life social networks and their activities that it replicates online (messages, meetings, news, etc). We are all the source of the training data. Why should we be at the whims of these derivative product companies?
- andreyk 6y ago
  Fun fact, OpenAI did not collect the data itself - they mainly used data from Common Crawl (in addition to a couple other datasets), which is compiled by a non profit that shares the dataset for free. So perhaps the license of such datasets can encourage free sharing of research outcomes. https://commoncrawl.org/ https://commoncrawl.org/
  - searchableguy 6y ago
    Shouldn't I own part of gpt3 because it's trained on my data and likely spitting out what I have commented somewhere on the internet? I think ml models should be public unless the data itself isn't.
    - searchableguy 6y ago
      Curious about the downvotes. If you remove compute and code, what remains is data that isn't owned by a single entity in the case of gpt3. You can sell both compute and code which is what you own but should you be able to sell data? You can't translate a commerical book without paying the copyright holder. You own the copyright for the translation but everything else still remains that of the original author. Why wouldn't this apply in case of a machine learning model trained on my images, messages, intents, likeness, etc?
      - isx726552 6y ago
        But you don’t understand, OpenAI has hype and “rockstars” and big names attached, so those rules don’t apply. Your data? It’s their data now. (And Microsoft’s.)
      - PeterisP 6y ago
        The current legal understanding is that according to the current copyright laws the ML models trained on some data are not considered derivative works of that data, so if the data was obtained legally and without other restrictions (e.g. if the researchers sign a contract with the data owner that gives them access to data but agree to certain conditions) then the authors of the data do not have any claim on some rights to the resulting model. One aspect of this is that historically statistical models calculated from large volumes of text (which is a notion that predates computers, e.g. frequency dictionaries and the whole [sub]field of quantitative corpus linguistics) have been considered facts about that corpus of text and thus not copyrightable at all or (depending on jurisdiction) entitled to different set of protections/limitations assigned to compilations of facts, which give some rights to the people who compiled the facts but no rights to the source of these facts (since facts as such can't be copyrighted). This also applies to many forms of analysis of audiovisual data, where the copyrights of the source works do not transfer to the results of the statistical or qualitative analysis and can't limit their creation, distribution or sale. The appropriate analogy to a commercial book or movie is not a translation, but some analysis of it - e.g. a thorough literary review and critique of some book or movie is a separate work with its own copyright, and the original author has no claim on it despite the fact that is (obviously) based on the contents of the work and describes it in great detail. Including verbatim fragments of the work is limited (fair use allows some inclusions but not all), but all the other details are not. Also, the whole notion of copyrightability of ML model weight files is interesting and IMHO not settled. You could argue that there is some creative expression in forming the model (which would support it being copyrightable) or you could argue that it's a mechanistic result of the application of some algorithm and settings (which have the creative part, and are copyrightable on their own), and so the output can't be copyrightable, no matter how much work (human or machine), time and cost it took - at least in USA copyright law doctrine (e.g. Feist Publications v. Rural Telephone Service) is that mere "sweat of the brow" (no matter how much) does not entitle a work to copyright protection; it requires application of human creativity to create an original work, and automated processes can't satisfy that requirement. So from the perspective of GPT-3 authors it might be prudent to distribute the model only to people who agree to specific, explicit contractual/licensing restrictions on redistribution and usage of the model instead of simply relying on the default application of copyright laws. And crucially, if some output is not copyrightable, it can't be considered a derived work to which the authors of a source work could have some claim.
whichquestion 6y ago
Would we be able to democratize these large ML models using something like SETI@Home or similar? Would more research in distributing models across large networks be useful? Has research on this topic been done?

ve55 6y ago

OpenAI's goal is definitely not to give everyone unlimited/equal access to powerful tools like GPT-3. We've had countless jokes about the name being 'OpenAI', and perhaps it's true that it's not the best name (along with 'democratizing' AI), but I'm not sure the author is suggesting a solution here rather than just venting that things seem kind of unfair, and no one outside of OpenAI really has much control or information available such as what he asks about. But I personally find the complaints to be understandable, especially as someone that didn't get a response for my requests for GPT-3 beta access, it felt pretty bad to watch everyone else have fun building cool things with the world's best text AI while I sat there and couldn't do anything, even if I was willing to pay for access. Hopefully there will be other relevant players here besides just OpenAI sooner or later.

langitbiru 6y ago
"other relevant players" -> Google could create one.
ma2rten 6y ago
The article is hinting at this but I also think many people who complain that OpenAI didn't release the model don't understand how big this model actually is. Even if they had access to the parameters they couldn't do much with it. Assuming you used single precision the model is 350 gigabytes (175 billion * 2 bytes). For fast inference the model needs to be in GPU memory. Most GPUs have 16GB of memory, so you would need 22 GPUs just to hold the model in memory and that doesn't even include the memory for activations. If you wanted to do fine tuning, you would need 3x as much memory for gradients and momentum.
- tarboreus 6y ago
  If it was open, there would be other services offering this, and not just an opaque beta and now a single expensive service.
- wslh 6y ago
  Tell cryptocurrency miners that this is a big model to compute... the size of this problem seems very tiny. If there are millions of ASIC, GPU, etc devices mining cryptocurrencies it is fair to speculate that democratizing AI has a special room in this model.

angel_j 6y ago

Forget AI democritization, GPT-3 is AI demoguerization. GPT-3 is singular; it is one model, one dataset, one training. Yet it will be the only one that will exist for quite some time (or by far the most available), and now it will underwrite productization and malfeasance, a la mode pay to play. For example, I recently read a paper supposedly written by a Chinese dissident virologist, which report was disseminated by a group with questionable membership. Most of the jargon in the report going over my head, I had to wonder if the otherwise convincing verbiage wasn't the work of GPT-3.

boulos 6y ago

Disclosure: I work on Google Cloud and have worked with the OpenAI folks on large models. This article mixes both “should research be open” and “is this work cheaply reproduced / accessible”: For smaller, open models: > The average person could not recreate models of this size from scratch, but the models can run on a single machine with a single GPU. but about GPT-3: > GPT-3 represents a new circumstance. For the first time, a model is so big it cannot be easily moved to another cloud and certainly does not run on a single computer with a single or small number of GPUs. Instead OpenAI is providing an API so that the model can be run on their cloud. While I’d quibble with “for the first time” (it’s easy to generate mega models! Plenty of distributed mesh tensorflow stuff does that, etc.), I don’t think this is any different than large physics simulations. Is it “wrong” to have some groups push the boundary of what’s possible with supercomputers? I certainly don’t think so. If anything, it shows what’s possible and others can do the valuable work of “miniaturization”. In this specific area, PRADO is a good example relative to BERT. For my historic area of ray tracing research, we did lots of things on an SGI Origin that let us “jump ahead a few years” versus what we could have done on any basic workstation. You could argue that it’s not academically interesting (“you just ran this really big because you had the budget / hardware, whatever”) and reject the paper. [Edit: I consider this kind of work interesting from a systems perspective, not “ML”, but it’s still interesting!] I don’t think it makes sense to suggest that we should hold back progress based on NSF grant funding or least common denominator computing resources. How would you decide what’s acceptable? Is a single A100 affordable? Only a T4? Only a laptop? tl;dr: it’s fine to argue about openness and democratization being hollow marketing words. I’m not sure I would conflate openness with “everyone, everywhere should be able to run any scientific work without expense”.

robmsmt 6y ago
how big do the models get? are you able to say?
- boulos 6y ago
  As big as you want, kind of. The challenge, as in large-scale physics, is how many nodes you can stick together with sufficiently high bandwidth (low latency is less important in the ML space, because there are lots of ops per byte, unlike some CFD Simulations that have very few per update). On our Cloud TPU product page [1], we have a single TPU v3 pod with 32 TB of memory. For the most recent MLperf submission, the TPU folks hooked up four of them [2]. There’s obviously a reduction in scalability from doing so (see weak scaling versus strong scaling terminology), but that’s the interesting co-design question: what kind of models can you usefully train in an “even more distributed” mode? Outside of TPUs though, even our single 16x A100 offering has 640 GB all connected by NVLINK (other providers went with 8, so 320 GB of “system memory”) and there are at least a few in a single rack. So the era of TiB scale models is certainly “semi feasible” and “open to all”. The challenge is that you need to also train these for quite some time. 1000 V100s would cost you at least $2000/hr to rent. Many models are sufficiently complicated (not just large) that you end up training them for days and weeks, even with this much compute. So the numbers add up quickly. But just being “big” doesn’t mean “trained for a month on a supercomputer”. [1] https://cloud.google.com/tpu https://cloud.google.com/tpu [2] https://www.google.com/amp/s/cloudblog.withgoogle.com/products/ai-machine-learning/google-breaks-ai-performance-records-in-mlperf-with-worlds-fastest-training-supercomputer/amp/ https://www.google.com/amp/s/cloudblog.withgoogle.com/produc...

29athrowaway 6y ago

Powerful models are trained using large amounts of data that most people do not have access to. They are also trained powerful infrastructure that most people do not have access to. So, to speak of democratization is interpreting the current state of affairs incorrectly.

boulos 6y ago
See my sibling comment, but the dataset that the GPT-2 folks did was “just grab whatever people on reddit upvoted”. Any grad student might have done the same (and in a sense, James Hays and Alyosha Efros did something like that long ago with flickr data for im2gps). While GPT-3 is trained on a massive V100 cluster, you could probably do so with a much smaller one / there exist interesting smaller models. It’s expensive to rent this class of equipment, but it is available. The distinction is that OpenAI made a focused bet. Most research funding and labs spread their bets heavily (e.g., each institution or researcher gets $50k/yr of funding). OpenAI takes a different stance, but it’s not clear that they’re spending even as much as say Google, Facebook or other large institutions. It’s also not obvious that you even have to play the same game to get similar results.
mirekrusin 6y ago
They trained it on freely available datasets managed by nonprofits as somebody mentioned. Powerfull infrastructure can be democratized in similar way that SETI@home, cryptocurriencies and other P2P projects do it. In theory this can be done.

skybrian 6y ago

It’s certainly annoying not to have access to the API to try a few things. However, optimization has just begun and it seems like there will probably be competing models 100x smaller in a year or two?

m1rr0rman 6y ago

RussianNationalAnthem.mp3

mensetmanusman 6y ago

I learned some things. GTP-3 is so complex the model requires large cloud computing resources to run. Ergo, it is also very expensive to run. Assumption: Bleeding edge AI will require tens of millions of dollars of computation before new network architectures fall out of state space. After this, the models can be pruned to be ran by mere mortals. If this is true, OpenAI will not be able to move to the next level without partnering with very wealthy institutions a la Microsoft. If this is true, those calling for OpenAI to not monetize intermittent progress are essentially preventing next generation discovery, unless they have alternative monetization ideas to generate 8 figures for research.

choppaface 6y ago
There will always be models and research that require proprietary hardware and resources and thus cannot be “democratized.” PageRank required a ton of hardware and an index. Non-industry researchers will always be priced out of something. A key concern here with regards to the ethics-of-AI issue is that last year OpenAI refused to release GPT-2 because it was too ‘dangerous’. This year, GPT-3 is suddenly a revenue-generating Microsoft product. The Gradient article linked is one of the more diplomatic ways of calling BS on OpenAI’s strategy. Financial interests taint research in subtle ways, it has for decades, and OpenAI employees being paid $1m in cash salary need reminders of this fact. https://openai.com/blog/better-language-models/ https://openai.com/blog/better-language-models/
anoncareer0212 6y ago
Given the worst-case assumptions of people arguing the cloud is required[1], it's a one-time investment of $60,000, $200,000 in the absolute worst case. [1] https://news.ycombinator.com/item?id=24601264 https://news.ycombinator.com/item?id=24601264
- exged 6y ago
  That's for model inference. For training OpenAI said they used around 3000 PetaFLOPS / days on the largest GPT-3 model. That translates to about 300 Nvidia A100 GPUs if you want training to finish in a month (any slower and your researchers are not going to be able to make much progress). A system like that would cost at least $5M, probably more like $10M.

deleted 6y ago

[deleted]

cblconfederate 6y ago

I don't see where is the monetization potential of GPT3. Creating more spam? Making poetry in bulk? Being SOTA in language tasks doesnt make it useful per se, as NLP measures are rather abstract anyway. As for its technology, it's huge but unless they are hiding some secret sauce, it's a straigtforwardly transformer-based, which means someone somewhere with cheap electricity is already training it on a dump of the entire web. Where's the moat here? Perhaps in some future version that they 'll keep secret? Well it will be a sad day for humanity if they keep its development in secret, given how much its development has benefited from open science around the world.

chime 6y ago

Not saying I am a fan of everything going on in tech world but it is pretty evident that AI is going to end up happening the West World “Rehoboam” big AI brain way than the Jetsons AI maid way, not purely due to corporate interests but rather due to the inherent cost of just making it work. Unless we can have PB/EB worth of storage locally then it’s going to be the cloud. And a cloud API with 1000 EB storage will always outperform your local 10 PB AI model. The most practical approach that I can see would be to minimize the cost of accessing the model, making it free like author suggests for research, students, non-profits etc. and charging more for commercial usage, basically extending the existing cloud model. One thing that I think should be free for everyone would be testing the model for biases. Basically if every AI API to check bias on any topic was free, then it could be improved by anyone including marginalized groups. If GPT-3 thinks all Indians are either doctors, coders, or gas station owners then I would like to be able to test, verify, and offer a patch without any cost, maybe even a reward. Otherwise GPT-4-5-6 will end up throwing away all Indian sounding last names applying for a construction job.

jackcosgrove 6y ago

The PC was supposed to democratize computing. It did, and then Microsoft found a choke point. The internet was supposed to democratize communication. It did, and then Google found a choke point. It seems like this two steps forward, one step back pattern might be the rule rather than the exception. Even the article defines AI democratization in terms of using models rather than training models, as the costs of training sophisticated models seems beyond the common developer even according to idealists. AI in particular seems to be a centralizing technology at this point, given that a model is often a black box to the user. The amount of data as well requires a massive telemetry apparatus, and designing likely models seems the province of people with PhDs. So yes I'm a bit pessimistic that AI will have a democratizing effect on technology or society, at least in the near future.

srtjstjsj 6y ago
> The PC was supposed to democratize computing It was?
- DSingularity 6y ago
  Yes. Before that it was mainframes and those who had access to $ who had access to computing.
- nl 6y ago
  Yes. The name gives it away. And Microsoft's mission for many years was "a computer in every home".
  - pixl97 6y ago
    "One ring to rule them all, and in the darkness bind them."
decasteve 6y ago
The airplane and the radio brought people closer. It also improved the coordination and delivery of mass destruction. Technology can be a rollercoaster ride of benefit and detriment.

sillysaurusx 6y ago

I’m amazed no one pointed out that OpenAI will be dead within 5 years unless it monetizes GPT-3. Democratizing AI is a fine goal, but it’s secondary to not dying. At least, for OpenAI it is. And that should be obvious.

Fordec 6y ago

I applied for API access months ago and still haven't been able to tinker with it. Until it's immediate sign up and not some insider walled garden waitlist, from a developer perspective, OpenAI and GPT-3 is anything but Open. If I was overly pushed to get access, my best chance of success right now appears to be a search on Github for somebody elses leaked API key in a repo somewhere.

nl 6y ago

This a bad take. In 2 years time there will be a quantized, TensorCore optimised equivalent that will run fine on desktop GPUs. We've seen this over and over again and there is no reason to believe this is different.

alphachloride 6y ago

The models are so unwieldy that had the big companies not handled the training and infrastructure, no one would be able to explore these models. We would have had no democratization. Are these big firms are beholden to open up their work to the public for this utopian ideal of democratization?

bionhoward 6y ago

Exclusive licenses for MSFT to tech invented at GOOG isn’t “democratization”

scoot_718 6y ago

Paying for the API is also bad if you wanted to do literally anything with the raw outputs. Have an idea you want to try out on how to sample the text from the model? Tough. Pay your money and get our output. It's useless.