This is actually something people used to do. old ASR systems (even models like Wav2vec) were usually combined with a language model. It wasn't a large languag

This is actually something people used to do. old ASR systems (even models like Wav2vec) were usually combined with a language model. It wasn't a large language model, those didn't exist at the time, it was usually something based on n-grams.