The basic strategy is always the same: you use a dictionary to identify possible words, and if there's overlap use a language model to determine the most likely

The basic strategy is always the same: you use a dictionary to identify possible words, and if there's overlap use a language model to determine the most likely segmentation. The details can be very different, e.g. Fugashi is a wrapper for MeCab https://taku910.github.io/mecab/ https://taku910.github.io/mecab/ which uses conditional random fields, while the most popular Chinese tokenizer is probably Jieba https://github.com/fxsjy/jieba https://github.com/fxsjy/jieba which uses a combination of word frequencies and hidden markov models. For German, the task is much easier, since most words are already separated by spaces, so only compound nouns might be problematic, but it's rare for those to have more than one possible segmentation. (There are of course exceptions, like e.g. Wachstube, which could be either Wach-Stube or Wachs-Tube. EDIT for non-German speakers: cognates are watch-stove = guardroom, wax-tube = tube of wax.)