3 ms·
They're produced from a tokenizer. Technically they could be anything (even raw bytes) but you get better results by choosing a better tokenization strategy.
by striking 1y ago
They're produced from a tokenizer. Technically they could be anything (even raw bytes) but you get better results by choosing a better tokenization strategy.