So something like having a smaller model generate embeddings, and use the embedding distance between the predicted text and the expected text for the loss compu

So something like having a smaller model generate embeddings, and use the embedding distance between the predicted text and the expected text for the loss computation? That sounds like it would be incredibly expensive. But maybe it allows you to fine tune on less data, offsetting the cost a bit (and allowing you to fine tune on just your very best data)