One can use the LLM to generate the label to distill a model to the desired precision, I used that approach and it worked quite well and the model runs locally,

One can use the LLM to generate the label to distill a model to the desired precision, I used that approach and it worked quite well and the model runs locally, including creating the sentence embeddings, faster than the LLM, at a fraction of the cost. Now certain problem space may be large enough to require models where the runtime makes it non economical to run it locally, but Ml is still a game of heuristics, see each problem requires some experimentation.