3 ms·
My understanding is that sites like archive.org honor robots.txt retroactively not because they are required to, but to best honor the wishes of the content pro
by syncsynchalt 11y ago
My understanding is that sites like archive.org honor robots.txt retroactively not because they are required to, but to best honor the wishes of the content provider.