To begin with, the training material is tokenized and one thing the choice of the optimal tokenizer depends on is the source language(s). Consider that in Engli

To begin with, the training material is tokenized and one thing the choice of the optimal tokenizer depends on is the source language(s). Consider that in English, the optimal tokenization of "cats" may be "cat" + "s" (based on the vocabulary and morphology of English) but in another language that may make no sense. The choice of the tokenizer is probably more important in a language which has less training material available.