Tokenizing is just one very trivial step, and it is probably the simplest and least interesting part of the process. Embedding vectors are dramatically more int

Tokenizing is just one very trivial step, and it is probably the simplest and least interesting part of the process. Embedding vectors are dramatically more interesting and actually useful. There is a mad rush to write articles in the LLM / ML / AI space to show that you haven't been left behind (like a FOMO, but more a FO-looking-like-you-MO). Tokenizers are by far the easiest part of that stack to grok, so the end result are a seemingly infinite selection of tokenization submissions.