26 ms·

there’s another big problem that I read about a few days ago: many sites are blocking all crawlers that are not google through robots.txt file.