4 ms·Crawling robots.txt files is a great way to find fun stuff in general.by JonnieCache 15y agoCrawling robots.txt files is a great way to find fun stuff in general.