3 ms·
I agree. Perplexities (probability of a text) can be compared using different tokenization after normalization.
by kwrobel 7y ago
I agree. Perplexities (probability of a text) can be compared using different tokenization after normalization.
3 ms·