3 ms·
Actually, if you want to interpret robots.txt that way, it raises the problem of "how long can I consider a robots.txt valid for?"
by mmagin 13y ago
Actually, if you want to interpret robots.txt that way, it raises the problem of "how long can I consider a robots.txt valid for?"