I still use them. I had to make a few binary and multi class classification models in NLP and while Transformers models gave nice results, it was impossible for

I still use them. I had to make a few binary and multi class classification models in NLP and while Transformers models gave nice results, it was impossible for me to use them long term. Why? Because data were confidential, I could not use the Cloud to get a nice GPU, and I needed almost one model per enterprise. Training DistilBert for 3 epochs was like 12 hours and it was a domain where data/concept drift happen often which means re-training was mandatory on weekly basis, at least. After some feature engineering, I was able to get almost the same performance (<1% difference) with a model that was able to train in less than 5 minutes and could be use in production easily. I love the fact that SVM, at least in scikit-learn, have a built-in early stopping mechanism. Really handy.