Tokens are just integer numbers, showing their position in the big vocabulary - it's that simple :) And vocabulary is just an array / vector / list - it depend

Tokens are just integer numbers, showing their position in the big vocabulary - it's that simple :) And vocabulary is just an array / vector / list - it depends which programming language you use, each has each own terminology for that data structure. For example LLaMA vocabulary has 32,000 tokens.