4 ms·
Yes, it should stop recursive fetches. Furthermore, excessive unnecessary requests should also be stopped, although that is separate from robots.txt. At least,
by zzo38computer 1y ago
Yes, it should stop recursive fetches. Furthermore, excessive unnecessary requests should also be stopped, although that is separate from robots.txt. At least, these are what I intended, and possibly also you.