11 ms·
"We are releasing all of our models between 125M and 30B parameters, and will provide full research access to OPT-175B upon request. Access will be granted to a
by MasterScrat 4y ago
"We are releasing all of our models between
125M and 30B parameters, and will provide full
research access to OPT-175B upon request.
Access will be granted to academic researchers; those
affiliated with organizations in government, civil
society, and academia; and those in industry research laboratories."
GPT-3 Davinci ("the" GPT-3) is 175B.
The repository will be open "First thing in AM" (https://twitter.com/stephenroller/status/1521302841276645376 https://twitter.com/stephenroller/status/1521302841276645376):
https://github.com/facebookresearch/metaseq/ https://github.com/facebookresearch/metaseq/
- ALittleLight 4y agoI don't like "available on request". I just want to download it and see if I can get it to run and mess around with it a bit. Why do I have to request anything? And I'm not an academic or researcher, so will they accept my random request? I'm also curious to know what the minimum requirements are to get this to run in inference mode.
- javchz 4y agoMy bet it's probably a filter, trying to prevent create a even more realistic farmbots in social media, as they are already bad as they are now.
- robonerd 4y agoBut they'll consider requests from government and industry.. both greater threats in the information war than any private individual.
- xxpor 4y agoNot from their perspective
- robonerd 4y agoOf course. To somebody in Zuck's position, shoring up the power of the status quo is common sense.
- alimov 4y agoI don’t know whether this is true and have no way of knowing this with any degree of certainty, but to me it seems unlikely that Mark had anything to do with this stipulation (requesting access). Although it’s not unimaginable.
- IAmEveryone 4y agoSince “everyone” would include governments and industry as well, their restriction is guaranteed to not contain more bad actors than no restriction.
- bogwog 4y agoThat is dumb when you consider that this thing is likely going to leak anyways. It’s inevitable, and when it does happen, it will just end up in the hands of criminals/scammers and not the general public.
- londons_explore 4y agoIt's super easy to watermark weights for ML models. Just add a random 0.01 to a random weight anywhere in the network. It will have very little impact on the results, but will mean you can identify who leaked the weights.
- ipaddr 4y agoCompare two copies.
- vladf 4y agoSlightly modify a million random weights by changing the least significant bit up or down.
- ipaddr 4y agoCompare three copies.
- ALittleLight 4y agoOr slightly randomly modify all the parameters on the copy you distribute, then it will be a match for nobody.
- ipaddr 4y agoYou compare all three and average the variance of each value. So the more copies the better.
- ZephyrBlu 4y agoCouple of random ideas: - They are concerned about the usage of the largest model, so want to vet people - The 175B parameter model is so large that it doesn't play nice with GitHub or something along those lines
- metadat 4y agoEnding up in the wild is an eventuality, whether FB creates it or someone else, why draw it out? Bandwidth concerns is nonsensical these days, fb has nearly unlimited resources in that department. Set it free! It wants to be free.
- deleted 4y ago[deleted]
- jquery 4y agoThis is an ideal use case for a torrent.
- sanxiyn 4y ago"It wants to be free" is a ridiculous statement, considering that after full two years (GPT-3 was published in May 2020), there is no public release of anything comparable. In May 2020, was your estimate of time to public release of anything comparable shorter or longer than two years? I bet it was shorter.
- HWR_14 4y ago> "It wants to be free" is a ridiculous statement "It wants to be free" is based on the standard line "code/data wants to be free". It doesn't mean this cost nothing to produce or isn't valuable.
- londons_explore 4y agoIn big companies, something as simple as "host it on facebook.com/model.tar.gz" can be mountains of approval and paperwork.
- 4y ago
- dukeofdoom 4y agoTo prevent someone from building something that returns certain inferences that might be true but are politically taboo.
- bestcoder69 4y agoYou think GPT-3 generates text that's truthful? Have you used it even once?
- dukeofdoom 4y agoI haven't used GPT-3, but I did try out a site that was based on GPT2. I believe it was called "talk to transformer". But I never tried quarrying anything controversial. However, I bet this a concern and certain queries will be filtered or "corrected" to be more politically correct. To give you an example, a few days ago I made a comment one Alex Jones, and wanted to google him. The second link returned on him was from ADL. No way that's an organic result. So just curious, if you have access to GTP-3 what does it return on Alex Jones, or other queries like who runs the banks, or who owns the media, and so on.
- deleted 4y ago[deleted]
- CrispinS 4y ago> The second link returned on him was from ADL. No way that's an organic result. It might be, actually. I understand why you'd think that, but look at the results for other search engines. Kagi: ADL in 2nd place Bing: ADL in 3rd place Yandex: ADL not on the first page, but SPLC[1] is the the 6th result [1]: https://www.splcenter.org/fighting-hate/extremist-files/individual/alex-jones https://www.splcenter.org/fighting-hate/extremist-files/indi...
- dukeofdoom 4y agoThis logic kind of fails quickly. I bet you wouldn't use it to show that Tiananmen Square did not happen, by showing all Chinese Search Engine are in apparent agreement on it not happening.
- alar44 4y agoGimme gimme. I want all your research and man hours for free. Gimme gimme. They are a for profit company and don't need to release anything. It's not that hard to understand.
- ninjin 4y agoTrue; they are free to do as they see fit. But how about not leeching on the word “open” in that case? DeepMind is essentially the NSA (or Apple), OpenAI is paid-for cloud services with paper-based marketing, and FAIR may be the best of the bunch, but it still annoys the hell out of me that they push code with non-commercial clauses as their current default (these are legally complicated in a university context) and now a model that they label “open” despite not honouring the accepted meaning of the word. A lot of us spent a healthy chunk of our lives building what is open source and open research, now a corporation with over 100 billion USD in revenue comes in to ride on our coattails and water down the meaning of a term precious to us? How about you spend the time and money to build your own terminology? “Available”, perhaps?
- ALittleLight 4y agoSure, but I'm an individual and free to say what I do and don't like. Why is that hard to understand?
- acchow 4y agoI’m thankful they’re offering anything at all openly. Is it such a big deal a gigantic download is hidden behind a request form?
- HWR_14 4y ago> Why do I have to request anything? I'm guessing it could be one or a mix of these: They want to build a database of people interested in this and vetted by some other organization as worth hiring. Just more people to feed to their recruiters. To see the output of the work. While academics will credit their data sources, seeing "XXX from YYY" requested, and then later "YYY releases product that could be based on the model" is probably pretty valuable vs wondering which ML it was based on. A veneer of responsible use, maybe required by their privacy policy or just to avoid backlash about "giving people's data away".
- lumost 4y agoA 175 billion parameter model might be a couple hundred gigs on disk. The file is probably just too big for GitHub/other standard FB services.
- fxtentacle 4y agoI'd guess they want to limit traffic. Once Huggingface links to you, your bandwidth bill 100x-es.
- saynay 4y agoIf it is like many other models, part of the reason would just be to reduce their bandwidth costs. The models can be huge, and they want to limit those who just want to download it on a whim so they don't rack up $10k+ is bandwidth charges, as has happened to many others who hosted big models out on S3 or something.
- levesque 4y agoIf only there was a way to distribute large files in a peer-to-peer manner, thus reducing the load on facebook's servers to effectively nothing. That would likely result in a torrent of bits being shared without any issues!
- JackC 4y ago> Why do I have to request anything? And I'm not an academic or researcher, so will they accept my random request? Just a guess: you will have to contractually agree to some things in order to get the model; at a minimum, agree not to redistribute it, but probably also agree not to use it commercially. That means whatever commercial advantage there is to having a model this size isn't affected by this offer, which makes it lower stakes for Facebook to offer. And then the point of "academics and researchers" is to be a proxy for "people we trust to keep their promise because they have a clear usecase for non-commercial access to the model and a reputation to protect." They can also sue after the fact, but they'd rather not have to. Not saying any of this is good or bad, just an educated guess about why it works the way it does.
- JoeyBananas 4y agoA 175B parameter language model is going to be huge. You probably don't want the biggest model just for messing around.
- gwern 4y agoI expect they will release the models fully, perhaps even under nonrestrictive licenses. Most researchers aren't too happy about those sort of restrictions, and would know that it vitiates a lot of the value of OPT. They look like they are doing the same sort of thing OA did with GPT-2: a staggered release. (This also has the benefit of not needing all the legal & PR approvals done upfront all at once; and there can be a lot of paperwork there.)
- schleck8 4y agoRepo down?