3 ms·
The corpus was published in 2013. https://github.com/ciprian-chelba/1-billion-word-language-modeling-benchmark https://github.com/ciprian-chelba/1-billion-word-
by sp332 10y ago
The corpus was published in 2013. https://github.com/ciprian-chelba/1-billion-word-language-modeling-benchmark https://github.com/ciprian-chelba/1-billion-word-language-mo... (And it seems to only have 0.8 billion words?)
Edit: Here's one mentioned in one of the other papers, it has over 7 billion "tokens" which I think includes words and punctuation. https://ibm.ent.box.com/v/booktest-v1 https://ibm.ent.box.com/v/booktest-v1