3 ms·

LOL, I was reading the abstract and remembering there used to be a paper like that. Then I look at the title and see it was from 2020. For a moment I thought so

LOL, I was reading the abstract and remembering there used to be a paper like that. Then I look at the title and see it was from 2020. For a moment I thought someone plagiarised the original paper. Unfortunately BERT models are dead. Even the cross between BERT and GPT - the T5 architecture (encode-decoder) is rarely used. The issue with BERT is that you need to modify the network to adapt it to any task by creating a prediction head, while decoder models (GPT style) do every task with tokens and never need to modify the network. Their advantage is that they have a single format for everything. BERT's advantage is the bidirectional attention, but apparently large size decoders don't have an issue with unidirectionality.

jszymborski 2y ago
BERT and T5 models are slowly consuming the computational biology field, so they certainly aren't dead to all.
johnsutor 2y ago
BERT isn't dead for smaller tasks (think NER, Sentiment Analysis) where low latency is needed.
nickpsecurity 2y ago
There’s also articles for pre-training BERT models on hardware resources a small lab could afford. Those are still useful, too, even if not highly competitive. So, they could still have value for low-cost, small, model development.
hiddencost 2y ago
BERT is alive and well for most commercial uses of NLP. If you're running 100k QPS through the model with a budget of 0.1 cents per query, you aren't going to be using a GPT model for classification.
Tostino 2y ago
It helps that you can pretty easily frame a bidirectional task in a directional way. For example, fill in the middle tasks. You can have a bidirectional model directly fill in the middle... Or you could just frame that as a causal task by giving the decoder llm a command to fill in the blanks, and the entire document with the sections to fill replaced by a special token/identifier all as input, and the model is trained to output the middle sections along with their identifier. There we go, now we have a causal decoder transformer that can perform a traditionally bidirectional task.

jszymborski 2y ago

BERT and T5 models are slowly consuming the computational biology field, so they certainly aren't dead to all.

johnsutor 2y ago

BERT isn't dead for smaller tasks (think NER, Sentiment Analysis) where low latency is needed.

nickpsecurity 2y ago

There’s also articles for pre-training BERT models on hardware resources a small lab could afford. Those are still useful, too, even if not highly competitive. So, they could still have value for low-cost, small, model development.

hiddencost 2y ago

BERT is alive and well for most commercial uses of NLP. If you're running 100k QPS through the model with a budget of 0.1 cents per query, you aren't going to be using a GPT model for classification.

Tostino 2y ago

It helps that you can pretty easily frame a bidirectional task in a directional way. For example, fill in the middle tasks. You can have a bidirectional model directly fill in the middle... Or you could just frame that as a causal task by giving the decoder llm a command to fill in the blanks, and the entire document with the sections to fill replaced by a special token/identifier all as input, and the model is trained to output the middle sections along with their identifier. There we go, now we have a causal decoder transformer that can perform a traditionally bidirectional task.