3 ms·
French text is somewhere around 10-30% longer than the corresponding English text. I would guess much of what you save on smaller vocabulary is lost on the leng
by kardos 2mo ago
French text is somewhere around 10-30% longer than the corresponding English text. I would guess much of what you save on smaller vocabulary is lost on the lengthened text.
- ViscountPenguin 2mo agoWhat would matter more is the token length, depends how well your tokenizer was trained on french I guess.