Y
HN Search
Hacker News Search
new
|
comments
|
top
|
jobs
anas-awadalla
searching PlanetScale…
1.
▲
2.
▲
3.
▲
4.
▲
5.
▲
6.
▲
3 ms
·
1.
▲
by
anas-awadalla
2y ago
Hello! Totally agree that tokens will be model dependent. We chose to calculate tokens using the GPT-2 tokenizer as that is a common metric used by other datasets like fineweb. So this should roughly give you a sense of how large the data i
2.
▲
by
anas-awadalla
2y ago
Hello! Creator of MINT here. We do a lot of pre-processing of commoncrawl (which in its raw form isn’t all that useful for training models). This includes heuristics to remove low quality text and images and deduplicating documents, paragra