4 ms·
I feel like I’m taking crazy pills: Reddit is on CommonCrawl, which means its all available without API access, permanently from AWS’s CDN. These discussions ar
by civilitty 3y ago
I feel like I’m taking crazy pills: Reddit is on CommonCrawl, which means its all available without API access, permanently from AWS’s CDN. These discussions are completely moot because the path of least resistance was already available and in use by OpenAI
- coffeebeqn 3y agoThank you, I had no idea about that site
- mritchie712 3y agoGood call, but is upvote / downvote data in there? I'd imagine OpenAI is doing something with that.
- dmazin 3y agoI need to stress this more obviously in the article, but the goal is to protect Reddit's future data.
- civilitty 3y agoWhy would Reddit block CommonCrawl's access to future data?
- screye 3y agoYeah, all the publicly available reddit data is freely available and the cat has been out of the bag for a while. Reddit has a lot of other private information though. The entire corpus of moderated / censored comments is the internet's best alignment dataset. The real upvote/downvote numbers make for perfect RLHF. Hell, Reddit could remove ads and replace them with paid RLHFing bots. Charge OpenAI money to selectively deploy AI bots to certain QA subreddits. All AI generated comments get deleted within a week. That way it doesn't lead to comment spam, but the bot still gets amazing upvote/downvote feedback and a ton of direct comment-reply feedback. 3rd party apps can't 'ad block' it either, since only Reddit knows which comments are fake. (psst, reddit, if you're gonna steal my idea, I want to be paid for it)