They are automatically generated, the algorithms have a bunch of tricks, but essentially they merge together the most frequent token pairs until a desired fixed

They are automatically generated, the algorithms have a bunch of tricks, but essentially they merge together the most frequent token pairs until a desired fixed vocabulary size is reached. So, for example (looking at GPT-3 tokenizations - you can test them at, for example, https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer) "517" is a single token, but "917" is two tokens; and there's no explicit link whatsoever between the token "517" and tokens "5" and "17" other than what can be learned from data. This works well enough for almost all tasks, but fails in edge cases like when someone makes up a toy challenge that asks how many fives are in a large number.