3 ms·
What about extending robots.txt to include date ranges? For future domain-owners this is likely far too much control, but maybe that could be mitigated if IA t
by thisacctforreal 9y ago
What about extending robots.txt to include date ranges?
For future domain-owners this is likely far too much control, but maybe that could be mitigated if IA tracks DNS/whois/registration info too
- pimlottc 9y agoIf we did this, the date range would be set to "Forever" 99.99% of the time.