2 ms·
It depends on the intention. You should respect robots.txt for search indexing, for example, but not necessarily for something like archiving or creating altern
by paxys 7y ago
It depends on the intention. You should respect robots.txt for search indexing, for example, but not necessarily for something like archiving or creating alternative page layouts (e.g outline/reader view).