3 ms·
Kidding me? The average size of robots.txt is 500 bytes for one web site! Average web page size is 2Mb. So if a crawler respectfully downloads a robots.txt it c
by behindai 7y ago
Kidding me? The average size of robots.txt is 500 bytes for one web site! Average web page size is 2Mb. So if a crawler respectfully downloads a robots.txt it could significantly save more bandwidth not downloading protected pages.