Exactly, it reads single tokens in some places and then creates data from them. It uses the word token for that, but that's arbitrary. That's not what a Tokeni

Exactly, it reads single tokens in some places and then creates data from them. It uses the word token for that, but that's arbitrary. That's not what a Tokenizer does, which usually would create a stream of tokens for all the elements in the input. A token would be a string with some metadata.