2 ms·
There's a reason we still call the file robots.txt by that name, and not web-scraping.txt or search-engines.txt.
by dotancohen 1y ago
There's a reason we still call the file robots.txt by that name, and not web-scraping.txt or search-engines.txt.
- adobrawy 1y agoYea, the reason is RFC9309.
- dotancohen 1y agoThat's actually a good point, but we have RFCs superceding RFCs all the time. My favorite RFC, 8601, superseded I think half a dozen other RFCs.