15 ms·
Meta is inviting researchers to pick apart the flaws in its version of GPT-3
- lee101 4y agoHavn't got access to the OPT-175 models yet as they are prioritising researchers, bloom is going to be huge but these OPT models that they did release IMO are not a step forward from research done in GPT-NEO/EleutherAI and AI21. Its a great step forward it being shared/open but the models themselves i don't see having big impact. They just seem to loop around a lot i don't know why... i think they trained on datasets with duplicate content or lots of repeating characters. Check out https://text-generator.io https://text-generator.io which is orders of magnitude cheaper than GPT-3 and still makes creative writing/code autocomplete without too much looping issues that youd see with OPT models. For that repeating you can dial up the repetition penalty or N to generate more sequences in a single request (and you are only charged by the request not by characters/tokens which helps), often generating N results is much more creative than generating a long result as the generated output gets fed into the input when generating long text and often causes that kind of repetitive looping. Some tactics to mitigate that repetitiveness are: * Dont use OPT... * repetition_penalty/retries/seed * Generate N results and combine instead of doing one big generate as you dont know how many results you'll really get and they are less creative/likely contain shared info/repetitiveness. * Creative input prompts There's probably other creative ways of getting variety by splitting the generation into different calls with higher repetition penalty as you get longer or looping detection etc. Its easy to detect repetition in the structured case like chat but hard when doing longer text generation/creative writing
- option 4y ago175B is for research only and as far as I understood their ToU does not allow commercial usage. Currently, the largest LLM that is both free and commercially usable (Apache 2.0) is 100B YaLM from Yandex (russian’s copy of Google). However, they did not publish any details on their training data.
- plegresl 4y agoThe 176B parameter BLOOM model should be available soon: https://bigscience.notion.site/BLOOM-BigScience-176B-Model-ad073ca07cdf479398d5f95d88e218c4 https://bigscience.notion.site/BLOOM-BigScience-176B-Model-a...
- option 4y agoyes, looking forward to it especially because it is going to be multilingual by design
- charcircuit 4y ago>However, they did not publish any details on their training data. Yes they did. It's in the README.
- option 4y agoAll I can see is “ 1.7 TB of online texts, books, and countless other sources in both English and Russian.” if there are more details, can you please share a link? I am worried that “other sources” may contain Yandex.news which is a cesspool of anti-West and anti-Ukraine propaganda
- 533474 4y agoThe pile dataset is used for the English language
- remram 4y agoDirect link: https://github.com/yandex/YaLM-100B/blob/main/README.md#training-details https://github.com/yandex/YaLM-100B/blob/main/README.md#trai...
- timmg 4y agoDumb question: does 175B parameters mean the number of bytes (or floats?) in the model? Does that also mean you need the whole model in memory to do inference (in practice)? If so, not many machines have that much RAM. Makes it hard to "play" with.
- jackblemming 4y agoYandex and Facebook are both more open than OpenAI? And the world isn’t ending because large language models were released? Shocking.
- jacooper 4y agoOpenAI is basically only open in the name.
- enlyth 4y agoI guess "Closed source pay-as-you-go AI" didn't have quite a ring to it
- tiborsaas 4y agoMaybe they meant Opening up AI :)
- bobkazamakis 4y agoOpenWallet
- dqpb 4y agoIt’s basic run-of-the-mill gaslighting.
- thesiniot 4y agoIt reminds me of that time the US Air Force designed an "open source jet engine". https://www.wpafb.af.mil/News/Article-Display/Article/2011131/afrl-tests-in-house-rapidly-developed-small-engine/ https://www.wpafb.af.mil/News/Article-Display/Article/201113... Their definition of "open source" turned out to be: "the government owns the source IP instead of some defense contractor. No, you can't see it." In fairness, I'm impressed that they even got that far. How do you think the defense contractor lobbyists responded to that program?
- Judgmentality 4y agoYou dare question the gatekeepers of our future AI overlords?
- Tepix 4y agoMay 3rd, which is why Yandex's 100B model release is not mentioned.
- blip54321 4y agoOn the ethics front: * Yandex released everything as full open * Facebook released open with restrictions * OpenAI is completely non-transparent, and to add insult to injury, is trying to sell my own code back to me. It seems like OpenAI has outlived its founding purpose, and is now a get-rich-quick scheme. What I really want is a way to run these on a normal GPU, not one with 200GB of RAM. I'm okay with sloooow execution.
- TrinaryWorksToo 4y agoHave you looked into HuggingFace Accelerate? People have supposedly been able to make the tradeoff with that. Although you still need to download the huge models.
- leereeves 4y agoCan confirm. HuggingFace Accelerate's big model feature[1] has some limits, but it does work. I used it to run a 40GB model on a system with just 20GB of free RAM and a 10GB GPU. All I had to do was prepare the weights in the format Accelerate understands, then load the model with Accelerate. After that, all the rest of the model code worked without any changes. But it is incredibly slow. A 20 billion parameter model took about a half hour to respond to a prompt and generate 100 tokens. A 175 billion parameter model like Facebook's would probably take hours. 1: https://huggingface.co/docs/accelerate/big_modeling https://huggingface.co/docs/accelerate/big_modeling
- blip54321 4y agoThank you for the pointer. I've been poking at it with a fork for the past few hours, and realized I forgot to respond.
- chessgecko 4y agoThat already exists depending on your definition of slow. Just get a big ssd, use it as swap and run the model on cpu.
- leereeves 4y ago
- nharada 4y agoThe logbook is awesome: https://github.com/facebookresearch/metaseq/blob/main/projects/OPT/chronicles/OPT175B_Logbook.pdf?fbclid=IwAR1SYEN38IY9FvdnN2UQgCOLk0UrpQRj8eR5F6gsesSErC6rcQnCTDMz8GI https://github.com/facebookresearch/metaseq/blob/main/projec... This is the true secret sauce -- all the tricks on how to get these things to train properly that aren't really published.
- sp527 4y agoThis really does read like 'DevOps: Nightmare Edition' > CSP fat fingered and deleted our entire cluster when trying to replenish our buffer nodes Yikes
- zubspace 4y agoI have no knowledge of such things, but it seems they run Cuda jobs on about 150 nodes? But why do they have so many problems to keep this cluster stable? Network failures? Bad GPU's? Bad drivers? Bad software? Running fixmycloud and going after all those cryptic errors every day seems like a nightmare to me...
- semi-extrinsic 4y agoSeems kind of par for the course for an HPC cluster, no? It makes sense to think of these things like Formula 1 cars, they are trying to eke out the absolute maximum performance, and reliability suffers because of that. "Ordinary" cloud is more like a Toyota where you optimize for fuel economy and low maintenance.
- amelius 4y agoThis all makes me wonder: how reproducible is the final output model?
- dekhn 4y agothe details of that specific model they ended up with? Irreproducible, unless the system was carefully designed and every detail required to do a fully reproducible computation was recorded and replayed. But they could easily produce a bunch of models that all sort of end up in roughly the same place and perform the same, ideally reducing the number of things they needed to change ad-hoc during the training.
- mgraczyk 4y agoReally concerning to me that people find the luddite argument so persuasive and that it gets so much play in the press. The crux of the argument from the outside "ethicists" quoted in the article is something like. "This new piece of technology might be dangerous and we don't fully understand it, so we should not poke at it or allow people to study it." Maybe there's just something about my personality that is deeply at odds with this sentiment, but it's also about the lack of testable predictions coming from people like this. Their position could be taken about literally anything with the same logical justification. It's a political and emotional stance masquerading as a technical or scientific process.
- trention 4y agoHere is one prediction: Because of language models, the amount of fake news online will increase by an order of magnitude (at least) before this decade ends. And there is no interpretation of that development as anything else but a net negative. Another more broad prediction: In a decade, the overall influence of language models on our society will be universally seen as a net negative.
- xmprt 4y agoWhat if that's the push that brings people out into the physical world where they don't have to deal with all this crap online.
- carschno 4y agoSlightly less optimistic, but perhaps more realistic thought: what if that's the push that makes people validate their (internet) sources? Seems like it might become clear that random web pages are just automatically generated content. If you really want to learn something credible at all, you'll really have to be more specific about your sources than "the internet".
- ninkendo 4y agoYeah, that was gonna be my contrarian HN hot-take too. Basically if it becomes really obvious some day that basically all online news publication is noise written by computers, maybe people will stop actually trusting it so much?
- abrax3141 4y agoYet another confabulation generator with pretty good grammar.
- sudden_dystopia 4y agoDidn’t we already learn our “free” lesson from this company?
- jimsmart 4y agoLink to original blog post by Meta (3 May 2022) https://ai.facebook.com/blog/democratizing-access-to-large-scale-language-models-with-opt-175b/ https://ai.facebook.com/blog/democratizing-access-to-large-s...
- deleted 4y ago[deleted]
- 4oh9do 4y agoWhat does "inviting" mean? It sounds like it means "Facebook wants free labor instead of paying for formal and expensive security audits".
- whoisjuan 4y agoMeta/Facebook has given the world React, PyTorch, GraphQL, Jest, and other fantastic technologies, and you are just boiling down their open source efforts to "Facebook wanting free labor." Not everything in tech is a sinister capitalistic plot. Open Source and Open Research are truly one of the best ways to accelerate technology advancements, in particular software technology advancements.
- makz 4y agoBeta testing for free?
- bribri 4y agoSo happy more stuff like this is open. Kudos to Meta
- annadane 4y agoOh really? Now you invite researchers instead of shutting down legitimate projects to investigate your algorithms?
- slowhadoken 4y agoIs this an advertisement for developers to work on Facebook's AI for free or am I being cynical?
- charcircuit 4y agoNo, it means that researchers now can have access to Facebook's large language model. No one is forcing the researchers to do research using it.
- rexpop 4y agoOf course it is! That's the premise of every open-source initiative, too. It's not too cynical, it's plain economics. Pretty sure it's the explicit purpose, too. No one really thinks open-source sponsorships are charity, do they?
- ArrayBoundCheck 4y agoWhy is facebook using GPT-3? Generate fake content it wants to push out?
- bravura 4y agoThat access form was... refreshing. Here's why this matters to me, an independent researcher who wants to start publishing again. In 2008, Ronan Collobert and Jason Weston had published some work that made neural network training of word vector representations really fast. But only ML people read that paper. Yoshua Bengio and Lev-Arie Ratinov and I plugged old-school cluster based as well as fancy-but-uncool-and-icky neural network word representations into a variety of NLP models. It worked awesome. Before "transformers go brrrrrr" our paper told the NLP community, basically, self-supervised learning and neural networks go "brrrrrrr". People finally started paying attention in the language world, ML stopped being treated with suspicion and the field moved rapidly, our paper racked up 2700 cites and an ACL 10 Year "Test Of Time" award, and here we are. I don't work in a big research lab but I still publish. I pay for my GPUs the old fashioned way. You know, out of pocket. It took me ages to get access to GPT-3. Ilya was a colleague of mine, so I messaged him fb, but no dice. Why? I know I could pull other strings through my network but, like, really? Is this where we are right now? All I'm saying is: It's nice to fill out a form asking my intended use and my previously related publications, as a means of gatekeeping. The access process feels more transparent and principled. Or maybe I'm just being grouchy.
- O__________O 4y agoLink to the ACL 10 Year "Test Of Time" award paper mentioned above: Word Representations: A Simple and General Method for Semi-Supervised Learning https://aclanthology.org/P10-1040/ https://aclanthology.org/P10-1040/ (PDF link) https://aclanthology.org/P10-1040.pdf https://aclanthology.org/P10-1040.pdf
- ninjin 4y agoThank you Turian, your ACL 2010 paper [1] was what got me somewhat early into word representations when it appeared during the first year of my PhD and ultimately gave me a head start on “deep learning thinking” that I suspect helped me greatly on my path to a faculty position many years later. My own lab at the time was not really a hardcore machine learning lab, but we read “Natural Language Processing (Almost) from Scratch” as well. However, it really was your paper that became the gateway to representation learning (and ultimately deep learning) for myself and several other members of the group. [1]: https://aclanthology.org/P10-1040 https://aclanthology.org/P10-1040 I clearly remember Christopher Manning explicitly mentioning your paper at the first ever (to the best of my knowledge) deep learning tutorial at a natural language processing conference (EMNLP 2012), at the very end before we left the room, with something along the lines of: “If there is only one thing you take away from this tutorial: Read Turian’s paper and add word representations as features to your classifiers for whatever task you may have – it works!” For those less familiar with the history of things, this was prior to Mikolov’s word2vec that arrived in 2013, which apart from popularising neural vector representations (“king - man + woman ~= queen”) above all made fast training of these embeddings feasible – Wikipedia overnight really, compared to the days, weeks, or even months of training for both the cluster-based and neural-based alternatives that were used back then. Outstanding work and very much ahead of its time. If you are ever around London, my group runs one of the largest public natural language processing talk series in the UK [2] and we would be more than happy to host you if you would be up for example to give a talk on a retrospective on the early days of things up until now. [2]: https://www.meetup.com/UCL-Natural-Language-Processing-Meetup https://www.meetup.com/UCL-Natural-Language-Processing-Meetu...
- anothernewdude 4y agoHow much are they paying for this service?
- westoncb 4y agoThis makes for some pretty excellent counter-marketing against OpenAI: "so Meta's GTP-3 is open?" "correct" "and the original is not?" "correct" "and the original is made by 'OpenAI'?" "correct" "hmm"
- needle0 4y agoIs the word "GPT-3" now being used like a a generic term, simply referring to a large-enough language model? From the article, Meta's OPT model seems to be intentionally designed to match the capabilities of OpenAI's original GPT-3, but doesn't say anywhere that it shares any code lineage.
- nodja 4y agoThat's because the code is unimportant. GPT-2 code is very simple, I think the official release by openai is only about a couple hundred lines of code. The challenge of GPT-3 is the scale, the GPT-3 paper basically says "This is GPT-2, but we made changes in the model so we can run it on dozens of GPUs". The changes mostly don't matter, because if you had a big enough GPU (~1.5TB VRAM) you could just up the hyperparameters of GPT-2 and you'd end up with the same results. So what's novel about GPT-3 that warrants a new name? The discovery here is that after the models reach a certain size, it's able to do many tasks without any training. You can literally ask it to translate from one language to another, at it'll do a decent job at it, if you give it a few examples it'll work even better. Now that doesn't mean that GPT-3 is the final model, it's still not good enough for many tasks, for example copilot is based on GPT-3 but it was specifically fine-tuned for the task of auto-completing code. So yes, if you can figure out how to scale a GPT-2 model to 175B parameters, you have a GPT-3 clone.
- vegai_ 4y agoTranslation: Meta wants researchers to work free for them, just like all of their content creators are doing right now. That company (and all the other parasitic social media companies like them) needs to be taxed way more heavily than it is being right now, based on the amount of content the people of every nation is generating for them.
- rdedev 4y ago> Meta AI audited OPT to remove some harmful behaviors How is this usually done in practice ?
- crabbygrabby 4y agoAh yes please work for free so meta can generate news articles, chat bots, and better targeted advertisements on the internet for billions of dollars a year. Thanks meta...
- mrandish 4y agoWhile I don't care at all for Facebook and never go there, this is a solid move by Meta which I appreciate. So far, I've been kind of turned off by OpenAI's apparent attitude when it comes to openness around GPT3, DALL-E, etc.