4 ms·
How do you extract the possible tokens from the model weights?
by dangoodmanUT 2y ago
How do you extract the possible tokens from the model weights?
- janalsncm 2y agoModel weights contain a mapping between logit index and the string representation of that token.
- singularity2001 2y agoat OP: another one posted the link to the tokens on githup two hours ago it's not part of the model but of the pre-processing