I think berts are still used for stuff like embeddings (at least the best generic embedding model in my language is bert based), when you want to get a model th

I think berts are still used for stuff like embeddings (at least the best generic embedding model in my language is bert based), when you want to get a model that has basic semantics but needs to be fast.