7 ms·
I'm sure his group has done some rigorous research that I can't even understand. But in my experience, the few-shot learner attribute of GPT-3 makes it insanel
by neural_thing 6y ago
I'm sure his group has done some rigorous research that I can't even understand.
But in my experience, the few-shot learner attribute of GPT-3 makes it insanely useful. We have already found several use cases for it, one of which replaces 2 ML engineers.
Yes, it's not perfect, but it's pretty good at many things, and REALLY easy to use.
- capableweb 6y agoCan you go into more details where it's useful? As your comment here goes directly against what's argued in the linked Facebook post. Also, if you've found a use case where GPT-3 replaces real humans, what did those humans actually spend their time on? Seems like either you're over-hyping GPT-3, or under-hyping humanity
- neural_thing 6y agoThe humans spent their time building a hideously difficult classification model. Out of the box GPT-3 worked better than the result of a year of their work.
- blackbear_ 6y agoJust because many more humans spent many more years and many more $$$ building GPT-3 for your convenience.
- fastball 6y agoRight, but GPT-3 can be used generally. That's the difference. It scales because you don't need to build an entirely new model for each different use case. You just change the prelude and use it for something new.
- sillysaurusx 6y agoIt sounds like a big deal. What a tempting idea. And a colleague was mildly annoyed with me for how unimpressed I seemed. But you have to understand, the use cases you mention are shallow and limited. The heart of GPT, the fine-tuning, is gone. And it looks like even OpenAI gave up on letting users fine-tune, because it means they essentially do build an entirely new, expensive model for each use case. I wanted to make an HN Simulator, the way that https://www.reddit.com/r/SubSimulatorGPT2/ https://www.reddit.com/r/SubSimulatorGPT2/ works. But that's far beyond the capabilities of metalearning (the idea that you describe).
- fastball 6y agoI think the onus is on you to prove that the use cases are shallow and limited. I've seen GPT-3 already being used for diverse and interesting ideas that would not have occurred to me personally. However, even if they are, the point stands: currently, there are teams of people at companies all over the world tuning models for these shallow and limited use-cases. GPT-3 can replace them all, without OpenAI needing to invest another cent in training for a particular customer's use-case. That is in fact game-changing for the ML/DL world and current applications thereof. Is it AGI? Obviously not. But the vast majority of ML applications don't need to be.
- sillysaurusx 6y agoWhat other proof would you like, other than an example of what I wanted to do and can't? (https://www.reddit.com/r/SubSimulatorGPT2/ https://www.reddit.com/r/SubSimulatorGPT2/ but for HN.) For a more extensive rebuttal, I wrote one here. https://news.ycombinator.com/item?id=23346972 https://news.ycombinator.com/item?id=23346972 Though that was more a rebut of GPT in general as a path to AGI than metalearning in particular for generating memes.
- fastball 6y agoGPT-3 not being suitable for your particular use case does not mean that all use cases are shallow and limited? That being said, I'm not sure I understand why you can't use GPT-3 to make an HN simulator.
- Traubenfuchs 6y agoHow did they react to this as humans with human pride? Sounds painful.
- ozorOzora 6y agoAs GP's declarations are backed by air, we can speculate they are self-reported statements by people working on non-business-centric applications. edit: GP giving more downvotes than proofs
- Voloskaya 6y agoSo GPT-3 didn't replace your 2 ML engineers, OpenAI did. GPT-3 didn't build itself.
- computerphage 6y agoThe iPhone didn't replace your flip phone, apple did. The iPhone didn't build itself.
- Voloskaya 6y agoYes except they were saying the iPhone replaced Nokia's engineers. GPT-3 is not doing what the ML engineers were doing (building models), GPT-3 is the end goal. The company just decided to outsource the work to OpenAI and pay a monthly fee to them instead of salaries to their ML engineers. "We have already found several use cases for it, one of which replaces 2 ML engineers." -> Clearly makes it sounds like GPT-3 can do the job their ML engineers were doing.
- DavidSJ 6y agoFrom a business perspective, this is an irrelevant distinction. The requirement was satisfied in a different way, i.e. the engineers satisfying the requirement were replaced by GPT-3, the tool which satisfies the requirement. I think everyone understood that.
- Voloskaya 6y agoThe thread is not about business perspective, it's about the hype around what GPT-3 is and is not able to do. One thing GPT-3 is not able to do for example, is replacing 2 ML engineers to build a GPT-3 like model. But OpenAI can do that.
- dwaltrip 6y agoIt’s not clear what you are trying to say.
- ApicalDendrite 6y agoWere there any concerns about GPT-3's latency? It looks like it takes a long time for online use cases.
- landryl 6y agoThat's interesting, GPT-3 can do classification too? Or did I misunderstood and you meant your engineers used classification to build a language model that didn't perform as well as GPT-3 (which is less surprising indeed) ?
- wongarsu 6y agoGPT-3 can do classification. For example you can give it a prompt like "Hacker News is a website. Excel is a Windows program. Visual Studio is a Windows program. Safari is a Mac program. CPU-Z is", and even GPT2 will complete this with "a Windows program" (with GPT2 you need to try multiple times, discard useless results and average what's most common, but it works and is straight-forward to automate).
- srtjstjsj 6y agoWhy? I can replace an excellent furniture designer with a much cheaper off the shelf desk.
- digital-cygnet 6y agoI would be interested in hearing more about this, within the bounds of what you can share publicly. Most of the touted GPT-3 use cases I've seen to date have dried up or are still in limbo, so hearing about a real production use would be exciting!
- neural_thing 6y agoExtremely complex classification task is all I can say
- lavoiems 6y agoGPT-3 is a generative model, isn't it? Can you explain how you converted GPT-3 to a classification model?
- 0-_-0 6y agoYou can add new "heads" to GPT networks and train those heads to use GPT for new applications.
- mkolodny 6y agoNot with GPT-3. I believe only Microsoft is allowed to do that.
- picodguyo 6y agoThere are a couple of ways to do it. You can give it a prompt that shows examples of the classification and it mimics what it thinks is the correct behavior when you feed it new unclassified input. They also have a search endpoint that lets you do classification by giving it an input along with labels as the searchable documents and using the resulting semantic relevance scores.
- ivalm 6y agoIn general GPT3 is not SotA on (any?) classification task, did you just not have enough data to fine tune a discriminative transformer model? Inference should be cheaper with a smaller transformer/also less lock-in.
- nickwalton00 6y agoYep few shot learning is a game changer. You don't get perfect results but you can prototype all kinds of systems extremely fast.
- fludlight 6y ago“AI” replacing the jobs of AI engineers. But we were told it was only going to do that to blue collar work!
- whymauri 6y agoBecause most "AI engineering" has lost its meaning and is actually data analysis.
- aaaxyz 6y agoDid it ever have a single meaning? Every company I've been through had a different definition of what "AI engineering" should be
- disgruntledphd2 6y agoAI engineering: a means of extraction of venture capital from gullible investors :) Mostly sarcasm, mostly.
- wongarsu 6y agoBetter IDEs have saved countless hours. Saving hours is equivalent to replacing jobs, unless demand is elastic enough to fill that time. Most of the time we are lucky enough that demand at a given price point is much larger than supply, but this won't last forever.
- ardit33 6y agothat's not how it works... as it is not a zero sum (i.e. the work is not bounded/fixed but it increases). IDEs and higher level tools help engineers become more productive. They can do more, with less. This raises the bar on products, and the demand of customers for them (things are pretier, easier to use, etc..), which in turn creates more domains for software to be used, and more demand for engineers. Google "Induced Demand"
- jimbokun 6y ago
- sillysaurusx 6y agoAnd when OpenAI says that your two entirely valid use cases are a safety concern, and denies you api access, what will you do? Better keep those ML engineers handy. If you think this isn’t a concern, I’ve already seen it happen with my own eyes, rather than hearing about it second hand. They encouraged someone to make a writing tool. That someone then spent roughly six weeks prototyping, iterating, and giving constant feedback. All signals from OpenAI were “Yes, awesome!” Then one day they simply declined to let them ship. Anything. Anything even resembling “a tool to generate huge quantities of outputs.” Which was, you know, the whole point. You play, you pay. And I hope you’re ready to pay, because you won’t have your magical genie unless the magical genie’s caretakers believe you are sufficiently worthy. I cloned as much of OpenAI’s API as I could: https://twitter.com/theshawwn/status/1312299759592333318?s=21 https://twitter.com/theshawwn/status/1312299759592333318?s=2... All that’s left is to reproduce a substantially similar model. Which is doable, but will take time. In the meantime, tread carefully.
- turdnagel 6y agoJust so you know, for GPT-3, Microsoft is going to be the exclusive licensee of the API: https://blogs.microsoft.com/blog/2020/09/22/microsoft-teams-up-with-openai-to-exclusively-license-gpt-3-language-model/ https://blogs.microsoft.com/blog/2020/09/22/microsoft-teams-...
- nmfisher 6y agoWere you a beta user, or is this now open for public access?