Perhaps I'm mistaken because I thought blingfire was mostly a tokenizer. But now I realize it might also be a language model. How are tokens/phrases/documents

Perhaps I'm mistaken because I thought blingfire was mostly a tokenizer. But now I realize it might also be a language model. How are tokens/phrases/documents represented, computationaly? Theoretically, do they live in a vector space?