3 ms·
The training data argument is a red herring. It's not difficult to scrape all of that via web for training purposes.
by The_Colonel 3y ago
The training data argument is a red herring. It's not difficult to scrape all of that via web for training purposes.
- pbmonster 3y agoIt's a legitimacy issue. If you get the data through the API, you don't have to hide that you used it. Which might be a big advantage if you want to publish/sell what you've been doing.
- TeMPOraL 3y agoBut isn't it the case that all that could be scrapped from Reddit before ChatGPT came on-line already has been scrapped, and is publicly available? Anything post-ChatGPT is going to be increasingly worthless, as there's an unknown amount of LLM-generated content mixed in it, which is going to be wasteful, and potentially poisonous when used in training next-gen LLMs (some early studies already indicate that training LLMs on LLM output leads to visible degradation very quickly).
- CodesInChaos 3y agoYes, there is a publicly available 2TB dump containing every post/comment until Feb 2023.
- Applejinx 3y agoAstonishing that it's only 2 TB…
- bombcar 3y agoI thick Reddit management believes they’re able to close this barn door and that additional horses will appear because Reddit content will continue to remain high quality and LLM-free. Wow that sentence was hard to write with a straight face.
- rcxdude 3y agoIt's unlikely that the legal landscape will be such that you would need to hide this at all. It would require a reversal of precedent on the legality of scraping and that training data is ruled to not be fair use.
- isaacfung 3y agoThere are data you cant scrap. e.g. you dont know what posts they feed to each individual user(assuming they use personalized recommendation like other social platforms), you dont know which post is shown to a user and is not clicked on, you dont know the subreddits a user subscribes to, you dont know which user upvotes/downvotes which comments, you dont know the users' browsing pattern(which subs they check first, when does a user open reddit during the day) These
- The_Colonel 3y agoNone of this is available in the Reddit API (at least the one used by 3rd party apps).
- busymom0 3y agoYep. And if I remember correctly, data upto some recent year is also available on Google’s big query?