It is a unit of text the LLM processes. :-) Everyone uses (byte pair encoding)[https://en.wikipedia.org/wiki/Byte_pair_encoding https://en.wikipedia.org/wiki/B

It is a unit of text the LLM processes. :-) Everyone uses (byte pair encoding)[https://en.wikipedia.org/wiki/Byte_pair_encoding https://en.wikipedia.org/wiki/Byte_pair_encoding] to generate their tokens; the tokens are whatever emerge from this. They will typically correspond to the most common substrings in the training corpus in, handwaving a bit, a max-cover sense; it's an encoding which attempts to best compress the data the tokenizer was trained on.