6 ms·
A couple months ago, reddit started refusing requests from my web scraper. Figured out they started checking the user agent and refusing connections that didn't
by twistedanimator 14y ago
A couple months ago, reddit started refusing requests from my web scraper. Figured out they started checking the user agent and refusing connections that didn't look like they came from a user's browser. Unless I missed an announcement somewhere, it doesn't seem like they're overly friendly about allowing web scrapers.
- citricsquid 14y agoThey are fine with it as long as you abide to their terms, they have a subreddit dedicated to reddit development and the reddit api which has discussion of scraping: http://www.reddit.com/r/redditdev http://www.reddit.com/r/redditdev
- randomdrake 14y agoThey just don't want to be abused. "We're happy to have API clients, crawlers, scrapers, and Greasemonkey scripts, but they have to obey some rules:" https://github.com/reddit/reddit/wiki/API https://github.com/reddit/reddit/wiki/API
- deleted 14y ago[deleted]