4 ms·
On a similar note, tools like [grabsite](https://github.com/ArchiveTeam/grab-site https://github.com/ArchiveTeam/grab-site) wisely use robots.txt as a method of
by slang800 7y ago
On a similar note, tools like [grabsite](https://github.com/ArchiveTeam/grab-site https://github.com/ArchiveTeam/grab-site) wisely use robots.txt as a method of finding additional paths to archive when crawling sites.