2 ms·
PG talked about blocking all non-Google crawlers in an attempt to cut down on traffic ... not sure if he ever did or not
by andreyf 18y ago
PG talked about blocking all non-Google crawlers in an attempt to cut down on traffic ... not sure if he ever did or not
- haasted 18y agoThere doesn't seem to be anything about it in HN's robot.txt. Of course, it may have been implemented in other ways.
- andrewparker 18y agoWhat are those "x" URLs blocked in the robots.txt for HN?