3 ms·
It's a legitimacy issue. If you get the data through the API, you don't have to hide that you used it. Which might be a big advantage if you want to publish/sel
by pbmonster 3y ago
It's a legitimacy issue. If you get the data through the API, you don't have to hide that you used it. Which might be a big advantage if you want to publish/sell what you've been doing.
- TeMPOraL 3y agoBut isn't it the case that all that could be scrapped from Reddit before ChatGPT came on-line already has been scrapped, and is publicly available? Anything post-ChatGPT is going to be increasingly worthless, as there's an unknown amount of LLM-generated content mixed in it, which is going to be wasteful, and potentially poisonous when used in training next-gen LLMs (some early studies already indicate that training LLMs on LLM output leads to visible degradation very quickly).
- CodesInChaos 3y agoYes, there is a publicly available 2TB dump containing every post/comment until Feb 2023.
- Applejinx 3y agoAstonishing that it's only 2 TB…
- bombcar 3y agoI thick Reddit management believes they’re able to close this barn door and that additional horses will appear because Reddit content will continue to remain high quality and LLM-free. Wow that sentence was hard to write with a straight face.
- rcxdude 3y agoIt's unlikely that the legal landscape will be such that you would need to hide this at all. It would require a reversal of precedent on the legality of scraping and that training data is ruled to not be fair use.