2 ms·
Tokenizer benchmark comparing 16 language models pre-trained from scratch
- alecco 3y agoThe actual analysis: https://github.com/alasdairforsythe/tokenmonster/blob/main/benchmark/pretrain.md https://github.com/alasdairforsythe/tokenmonster/blob/main/b... > Summary of Findings: > - Comparable (50256-strict-nocapcode) TokenMonster vocabularies perform better than both GPT-2 Tokenizer and tiktoken p50k_base on all metrics. > - Optimal vocabulary size is 32,000. > - Simpler vocabularies converge faster but do not necessarily produce better results when converged. > - Higher compression (more chr/tok) does not negatively affect model quality alone. > - Vocabularies with multiple words per token have a 5% negative impact on SMLQA (Ground Truth) benchmark, but a 13% better chr/tok compression. > - Capcode takes longer to learn, but once the model has converged, does not appear to affect SMLQA (Ground Truth) or SQuAD (Data Extraction) benchmarks significantly in either direction. > - Validation loss and F1 score are both meaningless metrics when comparing different tokenizers. > - Flaws and complications in the tokenizer affect the model's ability to learn facts more than they affect its linguistic capability.