Y
HN Search
Hacker News Search
new
|
comments
|
top
|
jobs
Lelleander
searching PlanetScale…
1.
▲
2.
▲
3.
▲
4.
▲
5.
▲
6.
▲
3 ms
·
1.
▲
by
Lelleander
3y ago
Most LLMs are trained using subword-tokenization such as BPE (which is investigated in this post) and Sentencepiece. These algorithms minimize the number of tokens required for representing the training corpus. I.e. for a training set mostl