5 ms·
It's not that big even as a text dataset. Common Crawl weighs in at some 250+TB. Even if we assume just 1% of that web data is usable text (it's likely much mor
by lovelearning 6y ago
It's not that big even as a text dataset. Common Crawl weighs in at some 250+TB. Even if we assume just 1% of that web data is usable text (it's likely much more), it's still 2.5+TB.
https://en.wikipedia.org/wiki/Common_Crawl https://en.wikipedia.org/wiki/Common_Crawl