The integers represent a position within a vector of "all known tokens". Typically, following a simple bag-of-words approach, each position in the vector would

The integers represent a position within a vector of "all known tokens". Typically, following a simple bag-of-words approach, each position in the vector would be toggled to 1 or 0 based on the presence of a token in a given document. Since most vectors would be almost completely zeroed, the simpler way to represent these vectors is through a list of positions in the now abstracted vector, aka a sparse vector, ie a list of integers. In the case of more advanced language models like LLMs, a given token can be paired with many other features of the token (such as dependencies or parts-of-speech) to make an integer represent one of many permutations on the same word based on its usage.