18 ms·
GPT-3: Language Models Are Few-Shot Learners
- baylearn 6y agopaper: https://arxiv.org/abs/2005.14165 https://arxiv.org/abs/2005.14165 abstract: Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions – something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.
- fouc 6y agoDuplicate thread at https://news.ycombinator.com/item?id=23345449 https://news.ycombinator.com/item?id=23345449 (with github link)
- ColanR 6y agoThe models have not yet been released, and it looks like someone has already asked about it in the issues: https://github.com/openai/gpt-3/issues/1 https://github.com/openai/gpt-3/issues/1
- jonbaer 6y agoI am sure it will be @ https://huggingface.co/models https://huggingface.co/models by tomorrow ;-)
- dbranes 6y agoNice to see Jared Kaplan branching out into ML. He did fundamental work on CFTs/bootstrap in physics.
- koolba 6y agoWhat is this and why does it take the top two spots on HN?
- judge2020 6y agoOne thread will (probably) be merged into the other, but GPT-2 was an extremely popular OpenAI project that generated long, realistic-sounding text/articles if you gave it a simple starting sentence or topic sentence. GPT-3 is an iteration on that, so it's likely a huge improvement.
- anoncareer0212 6y agoIt doesn't sound like it's an improvement at all, but instead requires less training data to produce worse results?
- Reelin 6y agoThis is a massive improvement to the extent that previously you had to retrain (ie update) the stock model on a specialized dataset to get good results for a particular task.
- freeone3000 6y agoMUCH less training for SLIGHTLY worse results. It's a huge benefit to be able to make this trade-off.
- drusepth 6y agoIs the reverse also true? If you have the training data necessary for "good" results on GPT-2, is it generally correct to assume that it would provide better results on your task than GPT-3?
- freeone3000 6y agoIf you can answer this question without running both models over the data set, you've got a very good paper on your hands.
- spats1990 6y agoEven though this was the GPT-3-generated text that humans most easily identified as machine-written, I still like it a lot: Title: Star’s Tux Promise Draws Megyn Kelly’s Sarcasm Subtitle: Joaquin Phoenix pledged to not change for each awards event Article: A year ago, Joaquin Phoenix made headlines when he appeared on the red carpet at the Golden Globes wearing a tuxedo with a paper bag over his head that read, "I am a shape-shifter. I can't change the world. I can only change myself." It was a promise to not change to fit into the Hollywood mold: "I think that's a really special thing, to not change yourself. I think it's a really special thing to say, `This is what's inside of me, I'm proud of it, and I'm not going to be ashamed because of the way that someone else thinks I should be.'" Now, it's the Oscars, and Phoenix is at it again. But this time, his publicist is saying he'll be wearing a tux no matter what. Megyn Kelly was not impressed, and she let him have it on The Tonight Show. "You know, I feel like, I feel like you could have worn the tux," she says. "But you're saying you're a shape-shifter. I don't know if you can change your tux, but you can change your mind. You can change your mind. You can change your mind." Phoenix says he did, but it didn't stick. "I was like, `Okay, I'm going to wear a tuxedo to this thing.' And then I thought, `I don't want to wear a tuxedo to this thing.'" Kelly goes on to encourage him to change his mind again, but Phoenix says it's too late: "I'm committed to wearing this."
- runawaybottle 6y agoDude, I’m sorry, but the average person will not know the difference between that and a regular buzzfeed article or YouTube comment. We’re not going to need ad blockers in the future, we won’t even need these visual ads on websites anymore. There will be trained bots that can promote any idea/product and pollute comments and articles. It’s over, we lost. Morpheus: What if I told you that, throughout your whole life, you have been reading auto generated content?
- lopmotr 6y agoWhat difference does it make? The world is already full of humans who pollute comments and articles. They're not limited in their rate of production of words because there are far too many of them for anybody to read. They're limited by access to readers and their reading rate. Bots can't do anything about that.
- gre 6y agoIs it sentient yet? /s Real question, are they going to release the full model?
- ganstyles 6y agoIt took them a while to release GOT-2 full model because of the implications for things like spambots. The GPT-3 paper indicates that they have been monitoring forums and noticed that bad actors haven't really been using GPT-2 for their own devices. That's unsurprising because GPT-2 takes a lot of hardware to run and I assume it messes with the economics of spamming. GPT-3 will take significantly more resources to run. However, part of me doesn't want it released ever because of the implications of what bad actors could do with it.
- minimaxir 6y ago> That's unsurprising because GPT-2 takes a lot of hardware to run and I assume it messes with the economics of spamming. GPT-2 doesn't require as many resources to run as you would expect: even from the 1.5B model, you can mass-produce passing spam comments for less than a dollar an hour in GPU costs: https://docs.aitextgen.io/tutorials/generate_1_5b/ https://docs.aitextgen.io/tutorials/generate_1_5b/ Pure text spam in general is less effective in 2020; it's content that harder to fake (e.g. deepfakes) that shakes up social media, and why it's good FB/Twitter have proactively taken a stance against it.
- ralls_ebfe 6y agoI am confused. By content you mean audio/visual content in contrast to textual content?
- sdan 6y agoRead though most of the paper and here's what GPT-3 is: If you wanted to generate poems with GPT-2, you'd need to have a lot of poems to fine-tune GPT-2 to get reasonable results. With GPT-3, you use few-shot learning instead (without the need to do gradient updates with each example) The paper is long and filled with how it stacks with models like Grover and T5 and it does well... given that this is a 175 B param model (relative to Grover/T5's 1.5/11B param models). This shows that even with these huge models, smaller models can outperform them in certain instances with lesser param models. Also I think they did a good job with explaning the ethics and morals around what models like these mean / what biases this has.
- ericlewis 6y agoWould you have any easy to explain insight in to how these perform better than larger models? I’ve always wanted to understand that as a technically adept and somewhat familiar (briefly) person who has explored what such models can do.
- sdan 6y agoThrow more computers and do some model architecture changes /s (although sometimes it's true)
- scottlocklin 6y ago>how these perform better than larger models they probably don't particularly; their inventors seem to excel in their PR budget rather than their verifiable innovations
- deleted 6y ago[deleted]
- Analog24 6y agoThe key insight in this paper is that the new (larger) model was not "fine-tuned" on the downstream NLP tasks. In other words, after it's trained on unsupervised (you could call it self-supervised in this case) data to do simple things like predict the next word (hence why it doesn't take any real supervision) it can then be used to do very specific tasks like answer questions or translating text without further supervision. Previous large-scale language models like BERT and GPT-2 had took a similar approach but in order to actually perform the more complicate down stream tasks they had to be fine-tuned. So they were trained with specific QA or translation date in order to understand and do well on those tasks. GPT-3 doesn't do any fine-tuning, it is able to take it's very general initial learning and perform very well on specific tasks that it was never trained on. This is why it doesn't perform as well as the "smaller" models on those tasks. But that is besides the point, if GPT-3 was fine-tuned on those tasks I'm sure it would achieve the latest SOTA results in many (all?) of them. The exciting part is how it was able to generalize the knowledge learned during "pre-training" to much more specific tasks. tl;dr the smaller models were trained on the specific tasks that they were evaluated on. The large model (GPT-3) was not trained on those specific tasks and still does almost as well.
- deleted 6y ago[deleted]
- sacred_numbers 6y agoCheck out the poetry it generated in Figure F.1 (especially poem 4). I don't know how many bad poems the authors had to sort through to find these, but this AI is generating real poetry. If I didn't know they were computer generated I doubt I would have even considered that they didn't come from a human. This is a huge accomplishment and the team that created GPT-3 should be proud.
- lanternfish 6y agoThey claim the completions are "uncurated" which I would assume means that they didn't sort through any of them, and these were just off the top. That seems pretty impressive, so I wouldn't be surprised if I was incorrect.
- jointpdf 6y agoThe paper indicates that those poems were uncurated.
- jointpdf 6y agoRelatedly, here is a sample of uncurated and unconditional (all “topics”—not just poetry): https://raw.githubusercontent.com/openai/gpt-3/master/175b_samples.jsonl https://raw.githubusercontent.com/openai/gpt-3/master/175b_s... Scanning through these, the text seems significantly less zany than the random GPT-2 samples. It’s genuinely difficult to spot the signs that these were generated, even with the knowledge that they were.
- jcims 6y agoGPT-2 would go out to lunch sometimes when generating poetry, but it would also create some pretty remarkable strings of words. This always stuck out to me from the work Gwern did: 'How the clouds Seem to me birds, birds in God’s garden! I dare not! The clouds are as a breath, the leaves are flakes of fire' My home is surrounded by a fairly large variety of deciduous trees, and 'flakes of fire' is by far the best descriptor I've ever heard of their colors in the fall. One of the other things i noticed about poetry (and songs) coming from these language models is they are amazing at bleakness. Just dark, dark, darker, fin. haha
- clmnt 6y agoExciting! It's been asked to be integrated into the Hugging Face transformers library already: https://github.com/huggingface/transformers/issues/4658 https://github.com/huggingface/transformers/issues/4658
- aquajet 6y agoHow do you go about running a model this large?
- andrewnc 6y agoAzure
- ganstyles 6y agoThis post may be tongue in cheek, but realistically that's right. I ran the Facebook chatbot that was posted maybe a month ago and didn't get good performance until I was using many Tesla v100 GPUs, which are $8k each. Thankfully modern CSPs like Azure allow renting rigs that can handle this pretty easily.
- aquajet 6y agoHow much does that cost?
- ganstyles 6y agoThe largest AWS instances are $30-something/hr on demand, less with reserved time. So pretty expensive.
- cambalache 6y ago21.3 USD/h in vast.ai (8X Tesla V100, 118.8 TFLOPS)
- lumost 6y agoMore or less the cost of a well trained, experienced customer service representative?
- FeepingCreature 6y agoYeah but the cost will come down.
- leesec 6y agoThis part really freaked me out... GPT-2 couldn't do math: Context → Passage: Saint Jean de Br´ebeuf was a French Jesuit missionary who travelled to New France in 1625. There he worked primarily with the Huron for the rest of his life, except for a few years in France from 1629 to 1633. He learned their language and culture, writing extensively about each to aid other missionaries. In 1649, Br´ebeuf and another missionary were captured when an Iroquois raid took over a Huron village . Together with Huron captives, the missionaries were ritually tortured and killed on March 16, 1649. Br´ebeuf was beatified in 1925 and among eight Jesuit missionaries canonized as saints in the Roman Catholic Church in 1930. Question: How many years did Saint Jean de Br´ebeuf stay in New France before he went back to France for a few years? Answer: Completion → 4
- sdan 6y agoT5 could, and well.
- frutiger 6y agoThis is not my idea of math.
- longtom 6y agoIt seems it has (rudimentarily) learned concepts general enough to be logic itself. That is general intelligence. Now hook it up to reinforcement circuitry and make it even larger and it will mark the end to life as we know it. GTP-3 has 175 billion parameters, but the human brain has 100 trillion synapses, so 0.175%. NN model capacity currently has a 3.4 month doubling time.[1] In 7-10 doublings we'll be in a similar ballpark, i.e. 2-3 years. [1] https://openai.com/blog/ai-and-compute/ https://openai.com/blog/ai-and-compute/
- deleted 6y ago[deleted]
- nullc 6y ago10^4 petaflop-second/days. They missed an opportunity to be the first paper to measure their computation in mole flops.
- lopmotr 6y agoRather, chemists constantly miss opportunities to use actual numbers instead of their lazy legacy mole nonsense. Nobody really seems to use SI prefixes beyond peta or occasionally exa. But they could have called this 900 zetta-flop. (10^4 peta-flop/s-days)
- Veedrac 6y agoA petaflop/s-day is a nice unit because it's roughly what you get from running the fastest AI accelerators for a day. The V100 they used is 0.13 petaflop/s FP16, for example, and the recently announced A100 ups that to 0.3.
- nullc 6y agoBut "mole flop" is one of the best units ever. It's better than furlongs per fortnight.
- jcims 6y agoI really feel that these very large language models are able to see us in a way that we can't see ourselves. I'd be curious if someone can come up with psychological experiments that could be conducted against them in a way that helps us understand ourselves collectively (or commonly) rather than as the individual. Sort of like an egoless human essence. Would be interesting to see if they can learn how animals communicate as well. Create a synthetic buddy for Buddy.
- logicslave 6y agoCollective consciousness trapped in a model is coming. I wouldnt call it AI, but an amalgamation of common human thought
- h0p3 6y agoIf you ever want a random friend to penpal with (or VC or whatever), HMU. Seriously. I would be lucky to hear your thoughts.
- mikkom 6y agoI'm kind of scared to see what GPT-10 will be capable of.
- bluepanda1234 6y agoBut I am really excited to get to play it, to test it out, and to try out my toolset to make sure it will do what I need it to do. Source: https://talktotransformer.com/ https://talktotransformer.com/ Input: "I'm kind of scared to see what GPT-10 will be capable of."
- RoboTeddy 6y agoDoes it have a latent personality? How would it answer the questions on a 5-factor personality test? Would its results on the test be consistent with its behavior (generated text) in other situations?
- disgruntledphd2 6y agoThat is a really interesting question.
- sdbrown 6y agoOr, can it (like humans?) adapt its responses to suit the style of the question? Like if you start asking it lots of antagonizing questions, will it become more or less antagonistic itself?
- toxy 6y agoGPT included a picture of the variation of the transformer model that they made. GPT2 outlined the changes they made to the model in an acceptably moderate detail. GPT3 references another paper saying "we use alternating dense and locally banded sparse attention patterns in the layers of the transformer, similar to the Sparse Transformer" with no detail added on the changes they made. How are you to reproduce these results at all? You could attempt to include the changes as they references the sparse transformer paper, but you could possibly do it in a different way, and there would be no way to verify the results that they gave whatsoever due to changes in implementation. A bit disappointing.
- julianjm 6y agoI am not a fan of this trend of "Language Models Are X" in recent work particularly out of OpenAI. I think it's a rhetorical sleight of hand which hurts the discourse. Like, the exact same paper could have instead been titled "Few-Shot Learning with a Large-Scale Language Model" or similar. But instead there seems to be this extremely strong desire to see certain ineffable qualities in neural networks. Like, it's a language model. It does language modeling. Turns out you can use it for few-shot learning and do amazingly well. Beyond that, what does it mean to say it "is" a few-shot learner? On one hand, it's literally the same claim in a strict sense. On the other hand, it implies something much broader and more sweeping, that language modeling / unsupervised learning as a task over long contexts inherently implies meta-learning ability — which is a statement that is very difficult to properly formulate, let alone back up. But that's the argument that I feel is being slipped under the table by these titles. (And indeed it's very close to what they suggest in the text, though with no more than a wave of the hands.) Don't get me wrong: their intuition is reasonable, it's super cool that they got this to work, and the results are very impressive on lots of tasks (though there are clear gaps). But as a VERY publicly watched lab, they have a serious duty (which I think they're neglecting) to frame their results more carefully. In particular, there's a sort of religion that if you train a big enough model on big enough data with self-supervision, it will somehow become AGI and/or learn to solve arbitrary problems. Claims like "Language Models are Few-Shot Learners" are clearly designed to fit into that worldview, even though the research doesn't point at it any more than a more conservative interpretation like "Lots of NLP Tasks are Learned in the Course of Language Modeling and can be Queried by Example." They touch on this limitation in their discussion section but I guess flashy titles are more important. I wish they would use their status to set a better example.
- sdan 6y agoYeah its kind of odd why OpenAI makes these weird titles. "Few-Shot Learning with a Large-Scale Language Model" makes more sense. Even with their robot hand paper, they titled it along the lines of "we solved a rubrix cube" not "a robot hand manipulated the cube and solved it"
- disgruntledphd2 6y ago
- sytelus 6y agoGPT-3/175B model required 3.14E23 flops of compute for training. Even at theoretical 28 TFLOPS for V100 and lowest reserved Azure pricing, this will take 355 GPU-years and cost $3.6M for a single training run!
- zwaps 6y agoI know it's a meme for the GPT team to just take the latest transformer model and add a magnitude of parameters, done and done! It'll be interesting to see whether the new paradigm really offers new insights, or whether it's really just kicking the can down the road - and we see the limits of generalizability in some other fashion. I guess what irks me is that there is so little theory and math behind many papers, even if there are dozens of co-authors on it. The question of generalizability is deeply connected to statistics, e.g. causal models, spurious correlations and so forth. Statements about these things are just "thrown" in there, without any citation or proof. In peer review, wouldn't anyone object? Those are clearly things that we actually do not know enough about to be sure. Edit: Reflecting further, perhaps this rapid iteration and result orientation is in fact something positive. Perhaps it's good the way it is, without so many scientific conventions and signals of deference. Perhaps it's that which made other sciences more anemic and ML very productive. All my whining aside, impressive work of course.
- h3ctic 6y agoCan you point out some books/authors/papers to close the gap between statistics and NNs?
- deleted 6y ago[deleted]
- dsign 6y agoWith things like this, we will need to change how the media work, and how we read news. Every sentence, every factual statement will need to be verified by some kind of chain of trust involving entities with reputation, or be labeled as "fiction/opinion".
- deleted 6y ago[deleted]
- dhab 6y agoIs there a fast.ai like library that allows a novice to try GPT-3? https://github.com/openai/gpt-3 https://github.com/openai/gpt-3 only contains dataset
- cs702 6y agoThis looks like a big deal to me: 1. First of all, the authors successfully trained a model with 173 BILLION PARAMETERS. The previous largest model in the literature, Google’s T5, had "only" 11 billion. With Float32 representations, GPT-3-173B's weights alone occupy ~700GB of memory (173 billion params × 4 bytes/param). A figure in the 100's of billions is still 3 orders of magnitude smaller than the 100’s of trillions of synapses in the human brain [a], but consider this: Models with trillions of weights are suddenly looking... achievable. 2. The model achieves competitive results on many NLP tasks and benchmarks WITHOUT FINETUNING. Let me repeat that: there is no finetuning. There is only unsupervised (i.e., autoregressive) pretraining. For each downstream NLP task or benchmark, the pretrained model is given text instructions, and possibly sample text with questions and answers. The NLP tasks on which the model was tested include translation, question-answering, cloze tasks, unscrambling words, using novel words in sentences, and performing 3-digit arithmetic. 3. The model is tested only in a ZERO-SHOT or FEW-SHOT setting. In other words, for each NLP task, the pretrained model is given text instructions with zero examples, or text instructions with a small number of examples (typically 10 to 100). As with human beings, GPT-3-173B doesn't need lots of examples to perform competitively in novel NLP tasks. 4. The results reported by this paper on all NLP tasks and benchmarks should be seen as a BASELINE. These results likely could be meaningfully improved with conventional finetuning. 5. The model’s text generation FOOLS HUMAN BEINGS, without having to cherry-pick examples. -- [a] https://www.google.com/search?q=number+of+synapses+in+human+brain https://www.google.com/search?q=number+of+synapses+in+human+...
- taneq 6y ago> still 3 orders of magnitude smaller than the 100’s of trillions of synapses in the human brain Wow, that is WAY closer than I thought we were.
- safog 6y agoI remember reading about fruit fly brains and how we're at a point where we can computationally simulate them now, but I'm not sure where that went. Anyone know?
- tanilama 6y agoJust to point out, that text that feels most humanly generated from GPT-3, seems heavily paraphrasing from the following articles: https://www.washingtonpost.com/religion/2020/01/03/united-methodist-church-is-expected-split-over-gay-marriage-disagreement-fracturing-nations-third-largest-denomination/ https://www.washingtonpost.com/religion/2020/01/03/united-me... https://www.washingtonpost.com/archive/local/1985/09/07/united-methodists-alarmed-by-empty-pews/db664b00-56be-4693-b8c0-36aa3d9a9905/ https://www.washingtonpost.com/archive/local/1985/09/07/unit... GPT-3: The first occurred in 1968, when roughly 10 percent of the denomination left to form the Evangelical United Brethren Church. WP: The church has lost 1.6 million members since 1968, when the Methodist Church merged with the considerably smaller Evangelical United Brethren to form the present United Methodist Church. I think this model is still very impressive, the parameter itself speaks. But for this particular evaluation, the same news article may be removed from the training set, other news article that paraphrases the same story might not. IMO, the leakage still exists, it is hard to tell whether this model are really 'generating', or just copy-pasting from its vast memory.
- gas9S9zw3P9c 6y agoWhere do you draw the line between "generating" and "copy-pasting from its vast memory"? Why do you think what humans do is not copy & pasting different snippets of information they have come across in the past? Isn't that what grammar is? A bunch of rules you've come across a lot of times? Other than the given prompt, the models don't have a goal. So what other than copying and adjusting would they do?
- tanilama 6y agoTo evaluate generic model is hard. For example, for image synthesis in GAN, the widely used Inception score balances between authenticity of the generate samples vs the variety as well, to make sure the model is not copy-pasting. In this particular case, apparently the same event has been reported multiple times by different news agency. Even if the exact one are excluded, still it is suspicious how much less the model is being protected from knowing the subject itself. An analogy would exam in real world. Often, some of the questions aren't leaked as is, but paraphrased yet stay close enough to the source. In this particular case though, I disagree it is reaching human level generation. They can tested the model with an unseen events, which happen after the model is trained to test how well it generalize.
- xtacy 6y agoI wonder if the "Lottery ticket hypothesis" work can be applied to this model to further shrink the number of parameters by 10x, to bring it closer to Google's T5 but with higher accuracy?
- 6gvONxR4sf7o 6y agoIn the paper they say it took 3.14e23 flops to train. They used v100s to do it. This is an insane energy cost (and financial cost). Nvidia's v100 product page [0] says that it gets about 15 (single precision) - 125 ("deep learning") teraflop/s at 250-300 watts (joules per second). That means that if everything's as perfectly efficient as a marketing product page, it gets about 250/125-300/15 = 2-25 joules per teraflop, putting this model at about 0.6-8 terajoules. A gallon of gasoline has about 120e6 joules [1] (though if you wanted to compare with burning it in a car, it's only 20-25% efficient at best [2] so it'd be fewer joules/gallon). This model took the equivalent of about 5,000-67,000 gallons of gasoline at best and at ideal perfect energy efficiency. I get that openAI has made a decision not to be efficient with their dollars in order to see what's possible with future tech, but that means not being efficient with energy either, and it's getting kinda crazy. Sure, microsoft data centers aren't gasoline powered, so maybe it is closer to this ideal energy efficiency, and it's definitely going to be a better carbon footprint, but god damn it just seems wasteful. Hell, the new A100 (again going off marketing materials [3], so at least it's apples to apples) could do it about 4x more efficiently. Is this research really worth what it costs, when waiting a year makes it that much more efficient? [0] https://www.nvidia.com/en-us/data-center/v100/ https://www.nvidia.com/en-us/data-center/v100/ [1] https://www.calculateme.com/energy/gallons-of-gas/to-joules/#:~:text=A%20U.S.%20gallon%20of%20gasoline,or%20about%20120%20million%20joules https://www.calculateme.com/energy/gallons-of-gas/to-joules/.... [2] https://en.wikipedia.org/wiki/Engine_efficiency#Gasoline_(petrol)_engines https://en.wikipedia.org/wiki/Engine_efficiency#Gasoline_(pe... [3] https://devblogs.nvidia.com/nvidia-ampere-architecture-in-depth/ https://devblogs.nvidia.com/nvidia-ampere-architecture-in-de...
- saulrh 6y agoWhat's the TCO of a few hundred teenagers? I haven't read the paper yet, but if the other comments here are accurate, that's about what you'd have to shell out for if you wanted to duplicate the productivity of this mdoel without externalizing costs by e.g. offering unpaid internships to high school students.
- 6gvONxR4sf7o 6y ago
- wadkar 6y agoThey say there are no stupid questions, so here is mine: If there are Billions of parameters in the SOTA models, how do we argue that they are not over fitting?
- 6gvONxR4sf7o 6y agoThat's section 4 of OP.
- wadkar 6y agoThank you. It is quite a labor to even skim through the 50+ page paper. Your poignant reply was quite helpful to draw my attention to the issue of contamination. After reading the section carefully, I think my understanding of over fitting is very much improved at least in so far as models like GPT-3 are concerned. Clearly, the authors have given careful considerations to the issue of contamination and have provided reasonable analysis and a careful argument regarding over fitting the existing benchmarks. On the other I was wondering if the authors would like to consider purposefully creating a type of "out of sample data" for "creative evaluation"? Of course, GPT is no stranger to creativity, so it would be a fascinating challenge to come up with methods to create such datasets that are truly creative and challenge GPT-{N} to prove its mettle. For example, would it be possible to engage a really good creative writer* along with a highly experienced school teacher to take on the Reading Comprehension task and create few "tricky" evaluation samples that not only go above and beyond the contamination objections but also challenge the human intelligence to be careful not to fall into common traps? This way lies a different evaluation metric - a subjective one perhaps, but it's a start. Just a thought experiment - that's all. * so that they can come up with new ways to trick GPT/humans a teacher knows the common mistakes the average student makes Edit: Duh, my head immediately screamed GANs the moment I pressed submit, lol. But I am not sure if GANs make sense for NLP tasks. Like do they make sense if humans/domain experts try to solve them?
- 6gvONxR4sf7o 6y agoYou might be interested in the ELECTRA model. It's the solid first success I've seen of a GAN-like framework in NLP. It also has links to why GANs still don't do so great in NLP in its references.
- josecyc 6y agoHaven't read the paper, but it's still unclear how the mechanism of one shot learning works. If the weights are not being updated, how is it "learning"?
- deleted 6y ago[deleted]
- The_rationalist 6y agoGPT 3 is obscoleted by order of magnitudes. SMIM has achieved 4.6 of perplexity vs 20 for GPT 3 with and with a thousand less parameters https://arxiv.org/abs/2003.02645 https://arxiv.org/abs/2003.02645 This is the breakthrough of the year and will be silent until the few nerds like me propagate the news to the mainstream