2 ms·
I'm unsure what you mean by "actual NLP practitioners". In any case does it really make a difference? We've been having GPT-3 related articles on HN and in the
by YeGoblynQueenne 6y ago
I'm unsure what you mean by "actual NLP practitioners". In any case does it really make a difference? We've been having GPT-3 related articles on HN and in the techie press for weeks after GPT-3 was published.
>> That's because they are literally different model architectures. The 3-gram model is the same everywhere, it doesn't depend on dataset. A 3-gram "trained" on Russian would still be a 3-gram. Similarly, BERT trained on a German corpus would still be BERT, it wouldn't need a "new catchy name." That's my point.
N-gram models are not "the same everywhere". An n-gram model is a set of probabilities assigned to, well, n-grams. Different sets of n-grams, or probabilities trained from different datasets would be- different. The n-grams would be different and the probabilities would be different and the structure of the model would be different. A Russian n-gram model would represent the probabilities of Russian n-grams, an English one would reprsent the probabilities of English n-grams and so on. The type of model would be the same, an n-gram model; but the model itself would be different. Otherwise, why bother training more models?
Same goes for a model trained with a transformer architecture. BERT is an English language model trained with a transformer architecture and representing the probabilities of tokens in an English corpus. A model trained with a transformer architecture and representing the probabilities of tokens in a German corpus would be, again, a different model. You could also call it "BERT" but that would eventually become very confusing.
>> Honestly, quite confused this is the hill you want to die on.
I'm not dying on any hill. Could you please not do that?
- MiroF 6y agoI disagree with how you define "model" to mean the parameters, rather than the form of the relationship between input and output. I don't think every time we make a new estimate of the electron mass, we are creating a "new" Standard Model. I don't really think it's super worth quibbling over, since I think we're generally on the same page. It's worth noting that BERT trained on different corpora is generally still called BERT (you can look at the huggingface/transformers library, for instance). I'm sorry that my message came across as more adversarial than intended.
- YeGoblynQueenne 6y ago>> I'm sorry that my message came across as more adversarial than intended. Thanks. I enjoyed our conversation.