One thing I'm not clear about is how the LLM decides on the correct tokenization to use. Looking at GPT2's token vocabulary, it has the word "the" as token num

One thing I'm not clear about is how the LLM decides on the correct tokenization to use. Looking at GPT2's token vocabulary, it has the word "the" as token number 1169, but it also has "t", "h", and "e" separately as tokens 83, 71, and 68. When the tokenizer is reading the prompt, does it greedily create the longest tokens possible from the start of the input, or is there a more sophisticated process going on? I guess in theory it could try multiple tokenizations and see which one results in the overall best response.