Very basic overview: A token is assigned a number, that number gets passed into the encoder model with other token numbers, and the encoder model transforms tho

Very basic overview: A token is assigned a number, that number gets passed into the encoder model with other token numbers, and the encoder model transforms those number sequences into embeddings (vectors)