2 ms·
Well Reddit would have a lot less scraper traffic if it hadn't shut down the feed API 3 years ago in an attempt to keep its data closed and only distributed to
by inigyou 4mo ago
Well Reddit would have a lot less scraper traffic if it hadn't shut down the feed API 3 years ago in an attempt to keep its data closed and only distributed to Google (which paid money for it). Self-inflicted. No sympathy.
- corbet 4mo agoI doubt it. The scraper botnets aren't going to bother with niceties like APIs, they just come in over HTTPS and grab anything they can.
- inigyou 4mo agoThere was a whole mini-industry around scraping specifically Reddit using the API feed because it was so open. AI companies first trained on Pushshift, which is like Common Crawl for Reddit, which is why Reddit shut down Pushshift (with legal threats IIRC) at the same time.
- Hizonner 4mo agoIf there were a well-known URL that retrieved all content added to a site after a given date, and a meaningful number of sites implemented it, I bet the biggest scrapers would use it. Except, of course, that then sites would start lying and sending abridged, poisoned, or nonexistent data. Nice things cannot be had, no matter how many levels you go down.
- inigyou 4mo agoReddit used to have something like that. And what you said happened. They shut it down, so that they could ask AI companies for payment.