4 ms·
If your robots.txt allows access to the user agent crawling (which Reddit does except for a very small denylist), and the crawler isn't making any attempt to by
by ryanbrunner 3y ago
If your robots.txt allows access to the user agent crawling (which Reddit does except for a very small denylist), and the crawler isn't making any attempt to bypass the site's security, they wouldn't have a leg to stand on legally.