3 ms·
We already have robots.txt in theory.
by loa_in_ 3y ago
We already have robots.txt in theory.
- layer8 3y agoI didn’t think robots.txt would be applicable to URLs being copied around, but actually it might be, good point. Though again, collecting that robots.txt information could make it easier to search for such URLs.