After training, tokens are vectors, but the number of unique vectors is limited by your vocabulary size (i.e. Should 'The' get a vector or should 'Th' and 'e' e

After training, tokens are vectors, but the number of unique vectors is limited by your vocabulary size (i.e. Should 'The' get a vector or should 'Th' and 'e' each get their own vector?). This step is deciding which clusters of letters (or whatever) get a vector and then giving them a scalar unique ID for conveniences' sake. The training then determines what that vector actually is.