3 ms·
I think you're misinterpreting the rule. The relevant robots.txt rules are: User-Agent: * Disallow: */comment/* Disallow: /r/*/comments/*/*/*/* T
by gruez 3y ago
I think you're misinterpreting the rule.
The relevant robots.txt rules are:
User-Agent: *
Disallow: */comment/*
Disallow: /r/*/comments/*/*/*/*
The url for the comment sections looks like: https://www.reddit.com/r/[subreddit]/comments/[id]/[slug]/ https://www.reddit.com/r/[subreddit]/comments/[id]/[slug]/. This doesn't match the above rules because there's only 2 parts after "comments", not 4 parts as specified by the rules.
- hollerith 3y agoNow it says "Disallow: /"