3 ms·
Probably HN is already part of The Pile [1]? I guess X is harder to scrap without permission. [1] https://pile.eleuther.ai https://pile.eleuther.ai
by nextos 3y ago
Probably HN is already part of The Pile [1]?
I guess X is harder to scrap without permission.
[1] https://pile.eleuther.ai https://pile.eleuther.ai
- capableweb 3y agoIt is indeed: We introduce new datasets derived from the fol- lowing sources: PubMed Central, ArXiv, GitHub, the FreeLaw Project, Stack Exchange, the US Patent and Trademark Office, PubMed, Ubuntu IRC, HackerNews, YouTube, PhilPapers, and NIH ExPorter. We also introduce OpenWebText2 and BookCorpus2, which are extensions of the original OpenWebText (Gokaslan and Cohen, 2019) and BookCorpus (Zhu et al., 2015; Kobayashi, 2018) datasets, respectively. From https://arxiv.org/abs/2101.00027 https://arxiv.org/abs/2101.00027 (The Pile: An 800GB Dataset of Diverse Text for Language Modeling)
- reaperducer 3y ago...HackerNews... And there's my incentive to stop posting on HN. It's been a blast, guys. I'm going back to lurker mode.
- JumpinJack_Cash 3y ago> > Probably HN is already part of The Pile Everybody smile for the camera, or we could just moon them, or both!