2 ms·
>> Why the hype? As is turns out, GPT-3 is unlike other natural language processing (NLP) systems, the latter of which often struggle with what comes comparativ
by YeGoblynQueenne 6y ago
>> Why the hype? As is turns out, GPT-3 is unlike other natural language processing (NLP) systems, the latter of which often struggle with what comes comparatively easily to humans: performing entirely new language tasks based on a few simple instructions and examples. Instead, NLP systems usually have to be pre-trained on a large corpus of text, and then fine-tuned in order to successfully perform a specific task. GPT-3, by contrast, does not require fine tuning of this kind: it seems to be able to perform a whole range of tasks reasonably well, from producing fiction, poetry, and press releases to functioning code, and from music, jokes, and technical manuals, to “news articles which human evaluators have difficulty distinguishing from articles written by humans”.
I have to stop at this before going on with the rest of the text. GPT-3 is, indeed, "pre-trained", on a huge corpus of unstructured text. The "range of tasks" in the passage above are actually all just one task, language generation. In fact, GPT-3 is specifically "pre-trained" on exactly that task, generating language. It doesn't need to be fine-tuned any further.
I think what the article is trying to say is that GPT-3 can perform some tasks that aren't usually thought of as strictly language generation tasks, such as machine translation or question answering, without having been specifically trained to do those things. "Not specifically trained to do those things" means that, e.g. instead of being trained with examples of sentences in one language and their translation in another, GPT-3 is trained with examples of arbitrary text respresenting a snapshot of many, diverse uses of language. Presumably, these diverse uses of language include translation, so GPT-3 also picked up the ability to do some translation. The same goes for question answerning etc. However, it should be noted that GPT-3 on the whole does not score very highly on metrics designed to measure performance on such let's say side-tasks. It certainly scores nowhere near as highly as specialised systems e.g. for machine translation etc. This is before looking at what those metrics actually try to measure which is usually only a very loosely defined sense of success [1].
Anyway I suppose I'll go on with the rest of the article, but I just wanted to flag this inconsistency up because it's a typical example of the subtle misunderstandings that circulate in the lay press about systems like GPT-3. Ultimately, for anyone who wishes to understand what systems like GPT-3 are, the best source is the relevant research and of course a bit of background in the subject and history of AI. If that sounds like a tall order, well tough. You can't replace knowledge of a field of research with more than 50 years of history with a quick read of five or six online articles and a dozen tweets.
_________________
[1] https://www.skynettoday.com/editorials/state_of_nmt https://www.skynettoday.com/editorials/state_of_nmt