2 ms·
I'd really like it if articles on scraping included robots.txt support.
by preinheimer 13y ago
I'd really like it if articles on scraping included robots.txt support.
- nashequilibrium 13y agorp = robotparser.RobotFileParser() rp.set_url("https://news.ycombinator.com/news/robots.txt" https://news.ycombinator.com/news/robots.txt") rp.read() # Reads the robots.txt rp.can_fetch("*", 'https://news.ycombinator.com/news' https://news.ycombinator.com/news') >>>> True
- NoodleIncident 13y agoThe robotparser library works fairly well. When I used it, it choked on a particular not-to-spec robots.txt file, but other than that it was fine. Copypasta of dead comment: nashequilibrium 13 hours ago | link [dead] rp = robotparser.RobotFileParser() rp.set_url("https://news.ycombinator.com/news/robots.txt" https://news.ycombinator.com/news/robots.txt") rp.read() # Reads the robots.txt rp.can_fetch("*", 'https://news.ycombinator.com/news' https://news.ycombinator.com/news') >>>> True