This is an awesome article, but I’m missing the part where solutions for each of the problems were discussed. Run a spell check before tokenizing? Maybe even t

This is an awesome article, but I’m missing the part where solutions for each of the problems were discussed. Run a spell check before tokenizing? Maybe even tokenize the misspelled word and the potential corrected word next to each other like „misspld (misspelled)“? For the issue with the brand names the tokenizer doesn’t know, I have no idea how to handle it. This problem is probably even worse in less common languages, or in languages which use a lot of compound words.