3 ms·
Anyone got a TL:DR? The crux of their argument isn't even on the first page...
by londons_explore 6y ago
Anyone got a TL:DR? The crux of their argument isn't even on the first page...
- Tepix 6y agoThey found a bunch of robots.txt files on sites like census.gov that blocked all robots except a few known ones, especially GoogleBot