10 ms·
AI training method exceeds GPT-3 performance with fewer parameters
- totorovirus 6y agoWell if an end task is constrained to one thing, then sure does a model with special tricks to optimize that goal performs better. GPT-3 is to confirm how deep and wide a language model can be to learn general task with few number of examples.
- 5986043handy 6y ago> although [the new technique] produced better results for NLP benchmarks, GPT-3 appeared more flexible.
- elephant_7 6y agoGPT-3 is also being used to make hedge funds money by generating investing themes https://medium.com/@492727ZED/dexamethasone-announcement-could-have-made-hedge-funds-a-fortune-alpha-week-313abb218a64 https://medium.com/@492727ZED/dexamethasone-announcement-cou...
- adamnemecek 6y agoI'm not surprised, a space with 175B parameters will be very sparse.
- raphlinus 6y agoIt's not obvious to me that this is the case. Chris Olah and others talk about "superposition" as a mechanism to explain "polysemantic" neurons that arise in image classifiers. To me, that suggests (using very vague, hand-wavy terms) that the optimization process is attempting to pack in as many concepts as possible into the finite parameter space. Certainly the scaling of GPT-3 suggests that these larger models are not necessarily any sparser than smaller ones. [1]: https://distill.pub/2020/circuits/zoom-in/ https://distill.pub/2020/circuits/zoom-in/
- ozorOzora 6y agoThat is fascinating, thanks for the link.
- lzzzfelipe 6y agoGPT3's strength is on language generation, so using *GLUE for evaluating it (where encoder type models are just better) and claiming to have 99.9% less parameters is sensationalism.
- visarga 6y agoYes, it can do simple yes/no questions but can it write dad jokes like GPT-3?
- dvfjsdhgfv 6y agoFor me another question is crucial: is it open (i.e. is there an open source reference implementation) or closed like the so-called "OpenAI" products/services?
- mkolodny 6y agoYou can find the code for the paper here: https://github.com/timoschick/pet https://github.com/timoschick/pet
- emteycz 6y agoCan you please explain to me (I don't understand it) why do you think GPT-3 is closed? Yes, they won't share the trained model, but they're sharing the research here[0][1] so you can reproduce easily, aren't they? As I understand it now, it's very fair - training the model is a separate thing from doing (and sharing) the research, is very costly, and would not happen if they were forced to open that too - I also don't understand why should they be. [0] https://arxiv.org/abs/2005.14165 https://arxiv.org/abs/2005.14165 [1] https://github.com/openai/gpt-3 https://github.com/openai/gpt-3
- dj_mc_merlin 6y ago> so you can reproduce easily That's like saying you can look at the Eiffel tower and it's schematics so what's so hard about getting a spare million dollars and building it.
- m3at 6y agoIf you want to read the source, I recommend skipping to their follow-up paper (same authors): https://arxiv.org/abs/2009.07118 https://arxiv.org/abs/2009.07118 Edit: tangentially related but for those who like to have a glance from their phone, arxiv-vanity is great instead of squinting at a pdf: https://www.arxiv-vanity.com/papers/2009.07118/ https://www.arxiv-vanity.com/papers/2009.07118/
- Jugurtha 6y agoNice. We're building a machine learning platform[0] where you can schedule a training notebook job, watch it on your phone. Examples[1][2]. Docs[3] - [0]: https://iko.ai https://iko.ai - [1]: https://twitter.com/jugurthahadjar/status/1307625017832869893 https://twitter.com/jugurthahadjar/status/130762501783286989... - [2]: https://twitter.com/jugurthahadjar/status/1313800728742449152 https://twitter.com/jugurthahadjar/status/131380072874244915... - [3]: https://iko.ai/docs/notebook/#long-running-notebooks https://iko.ai/docs/notebook/#long-running-notebooks
- Jugurtha 6y agoJust to clarify. This is about the fact it's cool to be able to read/consult what usually is better suited for a larger monitor, such as content from arxiv, or ML workloads. The links I shared are relevant because of that.
- throwawayiionqz 6y agoWhat is wrong with opening the PDF on a phone? Arxiv vanity has issues. Figure 2 in the paper you listed doesn't display correctly. I am not sure the author would agree or be happy that arxiv vanity reproduces their work into a webpage with sub-par rendering.
- mcdirty 6y agoI just wish I had the vocabulary to truly appreciate what's going on here.
- harwoodleon 6y agoDon’t worry, one day soon you’ll be able to ask an AI agent to explain to you what’s going on, in simple terms.
- nl 6y agoWhat a weird article. Semi-supervised training is always interesting, but leading with fact that it outperforms GPT-3 on superGLUE in the few shot setting where GPT-3 isn't training/fine-tuning but PET is. To make it clear: they finetune PET using synthetic data and then compare the results of this to GPT-3 where GPT-3 is initalized to prepare for querying. I would like to see how much overlap the synthetic data has with the queries used.
- jppope 6y agoSure they outperformed GTP-3... but can they DDoS my twitter account via a bunch of software engineers that won't stop talking about their model???
- mhuffman 6y ago... even more, is the new model "so goddamn powerful" that it can't even be trusted with normal humans, and must be doled out (for money) a little bit at a time so that some normie doesn't accidentally create skynet?
- panpanna 6y agoIt is starting to become a meme, but it sure achieved its PR goal?
- blt 6y agoWe should not be surprised to see new language models achieving GPT-3 performance with fewer parameters. The purpose (I assume, as an outsider) of the GPT-* project is to try to find the upper bound of how good a language model can be, without caring too much about efficiency. GPT-3 essentially scaled up the known good architecture of GPT-2. If you have a really big compute cluster, it makes sense to do experiments like this. It would be foolish to constantly try new methods without occasionally checking to see how far you can push current methods. A similar thing happened with VGGnet in image classification. It achieved SoTA with a huge amount of parameters, using the standard techniques of the time, but increasing the network depth. Later, people discovered a lot of tricks to get similar accuracy with fewer parameters.
- teruakohatu 6y ago> A similar thing happened with VGGnet in image classification I know I am at risk of a "Nobody will ever need more than 640KB of memory" comment, but the model size has exploded far in excess of our ability to improve GPU cards. GPT-3 is two orders of magnitude larger than VGGnet-16. Back when the VGG paper was published a Titan Z (2014 gen) had 12gb of RAM, while a Titan RTX (current/previous gen) has just doubled to 24gb.
- social_quotient 6y agoYour point is still valid but yesterday I learned about the newest workstation cards from nvidia - A6000 series with 48GB https://www.engadget.com/nvidia-rtx-a6000-a40-gpu-professional-144053827.html https://www.engadget.com/nvidia-rtx-a6000-a40-gpu-profession...
- jononor 6y agoWe can now each the performance of original VGG16/VGG19 with models that are much smaller. On ImageNet MobileNetsV2 matches the performance with 1/30 the size of weights, and same order of magnitude reduction in inference time and RAM usage. The same will probably happen with GPT-3. Of course we will probably also get models that surpass GPT-3 at the cost of even larger and more expensive models. The two are not at all exclusive.
- timkam 6y agoAm I the only one who finds it strange that a preprint gets that kind of news coverage and the authors of the preprint are the only ones who are interviewed? Even if it wasn't a preprint, I suppose it's good practice to ask independent experts for a second and third opinion. (Edit: a problem I see with such news coverage is that it really encourages researchers to rush their work out and make headline-grabbing claims even before they have any idea of what the scientific community thinks about the work.)
- antirez 6y agoYou do better than gpt3 with fewer parameters if you can have the same or better performances over the amount of tasks gpt3 can do well. To test a general model like gpt3 against something that can address a single problem does not make sense.
- yorwba 6y agoIt does make sense if your aim is to address that single problem. They're comparing against GPT-3 because that was the previous record holder (in the few-shot setting), despite being a more general model.
- desmap 6y agoSlightly OT: Will GPT-3 itself will have any impact IRL? I ask because of its lacking options/competition on the hosting side (there's just Azure). This implies high prices hindering most use cases to break even/be profitable.
- op03 6y agoMostly more effective spam. I see it as having access to the text generation capability of an infinite number of 15 year olds.
- mhh__ 6y agoI think it's already good enough to generate marketing copy (at a glance, not my area of expertise). Currently, AI is still very energy intensive though so it won't set the world on fire just yet - i.e. for many tasks the human brain still reigns king and it runs on something like 20W.
- scoot_718 6y agoGood, "Open-AI" can suck it, the pricks.
- buildbot 6y agoThis title is somewhat misleading, it’s more that you can use semi supervised learning to improve your fine tuning on a small model that’s already been pretrained. I think it’s impressive that this technique beats gpt-3 in any setting with such a small model, but for example, what stops you from applying this to gpt-3 itself and getting an equivalent gain? Saying that it exceeds gpt-3 performance sounds like it does better during pretraining, which would be extremely impressive.
- __tom__ 6y agoNLP is not "natural language processing" as mentioned in the article, it's "neuro linguistic programming".
- vaccinator 6y agoJust don't allow it to make itself better or we'll be in trouble....
- cs702 6y agoThe headline is clickbait (GPT-3 is about language modeling/generation, and there's ZERO mention of that in this work), but the actual work is interesting and the paper is worth a read. To the OP: please consider changing the headline to "Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference" and link to the original paper at https://arxiv.org/abs/2001.07676 https://arxiv.org/abs/2001.07676 instead of this PR puff piece.
- newen 6y agoNo one except people doing work on Cloze questions will be interested in this with that title. Badly written PR puff piece is bad but no one will read that abstract unless they are in that field.
- Voloskaya 6y ago> but no one will read that abstract unless they are in that field Which is fine? What's the point of getting people outside of the field reading a paper by essentially lying about the content? They come expecting A, and if they read it they understand it's actually about B. Loss of time for everyone except the author who wants to make a little buzz.
- newen 6y agoI'm just saying that there is a difference between a PR fluff piece that lies about the content and a PR fluff piece that tells the truth and makes research accessible or interesting to the lay public.
- msamwald 6y agoI think few-shot learning (or priming) is actually the main selling point of GPT-3 for most practical applications (rather than merely entertaining language generation). So if there is a method that achieves the same goal with a model that is simple enough to be used by normal developers and researchers without OpenAI-scale infrastructure, that does seem buzz-worthy.