I've wondered if this would happen, although doing inference directly on speech tokens would seem to imply an entirely different model (trained on lots and lots

I've wondered if this would happen, although doing inference directly on speech tokens would seem to imply an entirely different model (trained on lots and lots of actual speech).