In this paper https://arxiv.org/abs/2206.12693 https://arxiv.org/abs/2206.12693 I show that choosing the text snippets deliberately so that your choices minimiz

In this paper https://arxiv.org/abs/2206.12693 https://arxiv.org/abs/2206.12693 I show that choosing the text snippets deliberately so that your choices minimize overall entropy will also minimize result entropy, meaning it'll increase the accuracy of your speech recognition system (for German speech). In my case, the entropy-optimized tokenization also matched conserved features of the German language.