My line of thinking is using the more expensive model to label data, then use a teacher/student methodology to train the smaller model (SpaCy or BERT) for cost

My line of thinking is using the more expensive model to label data, then use a teacher/student methodology to train the smaller model (SpaCy or BERT) for cost & speed.