5 ms·

> If training a language model was "unremarkable" we wouldn't have a hypefest everytime anyone releases a new model, with a catchy name no less. BERT, ROBERTA,

by MiroF 6y ago

> If training a language model was "unremarkable" we wouldn't have a hypefest everytime anyone releases a new model, with a catchy name no less. BERT, ROBERTA, GPT-2/3/... etc. are remarkable enough for their creators to write papers exclusively for the purpose of announcing the model and describing the architecture used to train it etc. Actual NLP practitioners are not having a hype-fest every time someone spends a shitton to train a transformer LM on a new dataset. It's definitely cool, but I wouldn't cal it remarkable. What was cool about GPT-3 was the demonstration of the effectiveness of the "few-shot"/"conditioning" approach. I didn't find it surprising, but it was cool to push it to the next level. > In any case, I don't remember anyone ever naming their n-gram models or HMMs etc That's because they are literally different model architectures. The 3-gram model is the same everywhere, it doesn't depend on dataset. A 3-gram "trained" on Russian would still be a 3-gram. Similarly, BERT trained on a German corpus would still be BERT, it wouldn't need a "new catchy name." That's my point. As a side remark, you are vastly overestimating the amount of dataset specific tuning that you need - you can often just use the exact same hyperparams and run it on a new dataset - there's been some recent literature showing that a lot of these architectures are actually quite resilient to different datasets and hyperparameter choices and arrive at similar local minima. Sure, you might be off by 1-2% from SOTA accuracy (let's say on a classification task) but in production systems you're generally not shooting for the SOTA that was released in the last 2 months but rather something close-ish to those results. Honestly, quite confused this is the hill you want to die on.

YeGoblynQueenne 6y ago
I'm unsure what you mean by "actual NLP practitioners". In any case does it really make a difference? We've been having GPT-3 related articles on HN and in the techie press for weeks after GPT-3 was published. >> That's because they are literally different model architectures. The 3-gram model is the same everywhere, it doesn't depend on dataset. A 3-gram "trained" on Russian would still be a 3-gram. Similarly, BERT trained on a German corpus would still be BERT, it wouldn't need a "new catchy name." That's my point. N-gram models are not "the same everywhere". An n-gram model is a set of probabilities assigned to, well, n-grams. Different sets of n-grams, or probabilities trained from different datasets would be- different. The n-grams would be different and the probabilities would be different and the structure of the model would be different. A Russian n-gram model would represent the probabilities of Russian n-grams, an English one would reprsent the probabilities of English n-grams and so on. The type of model would be the same, an n-gram model; but the model itself would be different. Otherwise, why bother training more models? Same goes for a model trained with a transformer architecture. BERT is an English language model trained with a transformer architecture and representing the probabilities of tokens in an English corpus. A model trained with a transformer architecture and representing the probabilities of tokens in a German corpus would be, again, a different model. You could also call it "BERT" but that would eventually become very confusing. >> Honestly, quite confused this is the hill you want to die on. I'm not dying on any hill. Could you please not do that?
- MiroF 6y ago
  I disagree with how you define "model" to mean the parameters, rather than the form of the relationship between input and output. I don't think every time we make a new estimate of the electron mass, we are creating a "new" Standard Model. I don't really think it's super worth quibbling over, since I think we're generally on the same page. It's worth noting that BERT trained on different corpora is generally still called BERT (you can look at the huggingface/transformers library, for instance). I'm sorry that my message came across as more adversarial than intended.
  - YeGoblynQueenne 6y ago
    >> I'm sorry that my message came across as more adversarial than intended. Thanks. I enjoyed our conversation.

by MiroF 6y ago

YeGoblynQueenne 6y ago

I'm unsure what you mean by "actual NLP practitioners". In any case does it really make a difference? We've been having GPT-3 related articles on HN and in the techie press for weeks after GPT-3 was published. >> That's because they are literally different model architectures. The 3-gram model is the same everywhere, it doesn't depend on dataset. A 3-gram "trained" on Russian would still be a 3-gram. Similarly, BERT trained on a German corpus would still be BERT, it wouldn't need a "new catchy name." That's my point. N-gram models are not "the same everywhere". An n-gram model is a set of probabilities assigned to, well, n-grams. Different sets of n-grams, or probabilities trained from different datasets would be- different. The n-grams would be different and the probabilities would be different and the structure of the model would be different. A Russian n-gram model would represent the probabilities of Russian n-grams, an English one would reprsent the probabilities of English n-grams and so on. The type of model would be the same, an n-gram model; but the model itself would be different. Otherwise, why bother training more models? Same goes for a model trained with a transformer architecture. BERT is an English language model trained with a transformer architecture and representing the probabilities of tokens in an English corpus. A model trained with a transformer architecture and representing the probabilities of tokens in a German corpus would be, again, a different model. You could also call it "BERT" but that would eventually become very confusing. >> Honestly, quite confused this is the hill you want to die on. I'm not dying on any hill. Could you please not do that?

MiroF 6y ago
I disagree with how you define "model" to mean the parameters, rather than the form of the relationship between input and output. I don't think every time we make a new estimate of the electron mass, we are creating a "new" Standard Model. I don't really think it's super worth quibbling over, since I think we're generally on the same page. It's worth noting that BERT trained on different corpora is generally still called BERT (you can look at the huggingface/transformers library, for instance). I'm sorry that my message came across as more adversarial than intended.
- YeGoblynQueenne 6y ago
  >> I'm sorry that my message came across as more adversarial than intended. Thanks. I enjoyed our conversation.