3 ms·

This is very interesting, and I would be interested in learning more about the language-specific accommodations that are made. My minimal understanding of the G

This is very interesting, and I would be interested in learning more about the language-specific accommodations that are made. My minimal understanding of the GPT systems is that they are initially trained on a large corpus of English-language text, and then often, though not necessarily, given few-shot training on example tasks before being tested with questions that are scored. It would be unremarkable if the same system would have to be retrained from scratch to work with a different language, but I take it that you mean more than that. One guess is that the training data is annotated with grammatical information, in which case I would wonder if this is just a shortcut, or whether it solves a fundamental problem for such systems. Another guess is that the training set includes disambiguation, but that would seem to render meaningless the results on Winograd schema. (Update: I withdraw this last point, as presumably the Winograd-schema questions themselves are not disambiguated. Disambiguation of the training corpus would be a quite significant language-specific accommodation, if that is what is happening.)

MiroF 6y ago
> It would be unremarkable if the same system would have to be retrained from scratch to work with a different language, but I take it that you mean more than that. It really is that unremarkable, despite the GPs insinuations to the contrary.
- YeGoblynQueenne 6y ago
  If training a language model was "unremarkable" we wouldn't have a hypefest everytime anyone releases a new model, with a catchy name no less. BERT, ROBERTA, GPT-2/3/... etc. are remarkable enough for their creators to write papers exclusively for the purpose of announcing the model and describing the architecture used to train it etc. In any case, I don't remember anyone ever naming their n-gram models or HMMs etc. The reason for this of course is that training a language model with a deep neural network architecture is anything but unremarkable, not least because the costs in data, human-hours and compute are beyond the reach of most entities other than large corporate teams. If training a new model from scratch were truly "unremarkable" we'd all have our own.
  - mannykannot 6y ago
    You have hung a lot on the single word “unremarkable”, so perhaps I did not make it completely clear what I thought was unremarkable. The only thing that I meant was that if a given architecture for NLP would require retraining from scratch in order to work with a new language, I would not think that fact would be surprising, or be indicative of some fundamental flaw or limitation in the approach taken. If, on the other hand, a given architecture could not be retrained to perform well on a different language, or only on a group of related languages, that might be indicative of a fundamental limitation in how far it could be developed, depending on what sort of accommodations have to be made for new languages.
    - YeGoblynQueenne 6y ago
      Thanks for clarifying. Part of the reason why training a new model from scratch is so difficult is the amount of dataset-specific tweaking and fine-tuning that goes on and that is rarely discussed in publications. For example, papers willt describe the architecture of a system but will rarely go into much detail about how a particular component was selected over another (why this cost function over the other etc). This fine-tuning is essential to achieving state-of-the-art results and without it you can expect to produce a model that is simply not the same as the original. Certainly in terms of performance metrics. Basically retraining the same system on a different dataset is a bit like entering the same river in two points in time. Yes, in theory it's "the same" system and only the dataset chnges. In practice, it takes so much architectural tweaking and fine-tuning that it's a different system and you've done almost all the work from scratch. This also is part and parcel of why so much fanfare surrounds the publication of a new model. Because it's really, really hard to train a new model on a new dataset (assuming one wants to get state of the art results).
  - MiroF 6y ago
    > If training a language model was "unremarkable" we wouldn't have a hypefest everytime anyone releases a new model, with a catchy name no less. BERT, ROBERTA, GPT-2/3/... etc. are remarkable enough for their creators to write papers exclusively for the purpose of announcing the model and describing the architecture used to train it etc. Actual NLP practitioners are not having a hype-fest every time someone spends a shitton to train a transformer LM on a new dataset. It's definitely cool, but I wouldn't cal it remarkable. What was cool about GPT-3 was the demonstration of the effectiveness of the "few-shot"/"conditioning" approach. I didn't find it surprising, but it was cool to push it to the next level. > In any case, I don't remember anyone ever naming their n-gram models or HMMs etc That's because they are literally different model architectures. The 3-gram model is the same everywhere, it doesn't depend on dataset. A 3-gram "trained" on Russian would still be a 3-gram. Similarly, BERT trained on a German corpus would still be BERT, it wouldn't need a "new catchy name." That's my point. As a side remark, you are vastly overestimating the amount of dataset specific tuning that you need - you can often just use the exact same hyperparams and run it on a new dataset - there's been some recent literature showing that a lot of these architectures are actually quite resilient to different datasets and hyperparameter choices and arrive at similar local minima. Sure, you might be off by 1-2% from SOTA accuracy (let's say on a classification task) but in production systems you're generally not shooting for the SOTA that was released in the last 2 months but rather something close-ish to those results. Honestly, quite confused this is the hill you want to die on.

MiroF 6y ago

> It would be unremarkable if the same system would have to be retrained from scratch to work with a different language, but I take it that you mean more than that. It really is that unremarkable, despite the GPs insinuations to the contrary.

YeGoblynQueenne 6y ago
If training a language model was "unremarkable" we wouldn't have a hypefest everytime anyone releases a new model, with a catchy name no less. BERT, ROBERTA, GPT-2/3/... etc. are remarkable enough for their creators to write papers exclusively for the purpose of announcing the model and describing the architecture used to train it etc. In any case, I don't remember anyone ever naming their n-gram models or HMMs etc. The reason for this of course is that training a language model with a deep neural network architecture is anything but unremarkable, not least because the costs in data, human-hours and compute are beyond the reach of most entities other than large corporate teams. If training a new model from scratch were truly "unremarkable" we'd all have our own.
- mannykannot 6y ago
  You have hung a lot on the single word “unremarkable”, so perhaps I did not make it completely clear what I thought was unremarkable. The only thing that I meant was that if a given architecture for NLP would require retraining from scratch in order to work with a new language, I would not think that fact would be surprising, or be indicative of some fundamental flaw or limitation in the approach taken. If, on the other hand, a given architecture could not be retrained to perform well on a different language, or only on a group of related languages, that might be indicative of a fundamental limitation in how far it could be developed, depending on what sort of accommodations have to be made for new languages.
  - YeGoblynQueenne 6y ago
    Thanks for clarifying. Part of the reason why training a new model from scratch is so difficult is the amount of dataset-specific tweaking and fine-tuning that goes on and that is rarely discussed in publications. For example, papers willt describe the architecture of a system but will rarely go into much detail about how a particular component was selected over another (why this cost function over the other etc). This fine-tuning is essential to achieving state-of-the-art results and without it you can expect to produce a model that is simply not the same as the original. Certainly in terms of performance metrics. Basically retraining the same system on a different dataset is a bit like entering the same river in two points in time. Yes, in theory it's "the same" system and only the dataset chnges. In practice, it takes so much architectural tweaking and fine-tuning that it's a different system and you've done almost all the work from scratch. This also is part and parcel of why so much fanfare surrounds the publication of a new model. Because it's really, really hard to train a new model on a new dataset (assuming one wants to get state of the art results).
- MiroF 6y ago
  > If training a language model was "unremarkable" we wouldn't have a hypefest everytime anyone releases a new model, with a catchy name no less. BERT, ROBERTA, GPT-2/3/... etc. are remarkable enough for their creators to write papers exclusively for the purpose of announcing the model and describing the architecture used to train it etc. Actual NLP practitioners are not having a hype-fest every time someone spends a shitton to train a transformer LM on a new dataset. It's definitely cool, but I wouldn't cal it remarkable. What was cool about GPT-3 was the demonstration of the effectiveness of the "few-shot"/"conditioning" approach. I didn't find it surprising, but it was cool to push it to the next level. > In any case, I don't remember anyone ever naming their n-gram models or HMMs etc That's because they are literally different model architectures. The 3-gram model is the same everywhere, it doesn't depend on dataset. A 3-gram "trained" on Russian would still be a 3-gram. Similarly, BERT trained on a German corpus would still be BERT, it wouldn't need a "new catchy name." That's my point. As a side remark, you are vastly overestimating the amount of dataset specific tuning that you need - you can often just use the exact same hyperparams and run it on a new dataset - there's been some recent literature showing that a lot of these architectures are actually quite resilient to different datasets and hyperparameter choices and arrive at similar local minima. Sure, you might be off by 1-2% from SOTA accuracy (let's say on a classification task) but in production systems you're generally not shooting for the SOTA that was released in the last 2 months but rather something close-ish to those results. Honestly, quite confused this is the hill you want to die on.