3 ms·
The robotparser library works fairly well. When I used it, it choked on a particular not-to-spec robots.txt file, but other than that it was fine. Copypasta of
by NoodleIncident 13y ago
The robotparser library works fairly well. When I used it, it choked on a particular not-to-spec robots.txt file, but other than that it was fine.
Copypasta of dead comment:
nashequilibrium 13 hours ago | link [dead]
rp = robotparser.RobotFileParser()
rp.set_url("https://news.ycombinator.com/news/robots.txt" https://news.ycombinator.com/news/robots.txt")
rp.read()
# Reads the robots.txt
rp.can_fetch("*", 'https://news.ycombinator.com/news' https://news.ycombinator.com/news')
>>>> True