3 ms·
February 1994[1], so the 20 year anniversary of the robots.txt file. [1]http://en.wikipedia.org/wiki/Robots_exclusion_standard http://en.wikipedia.org/wiki/Rob
by sschueller 12y ago
February 1994[1], so the 20 year anniversary of the robots.txt file.
[1]http://en.wikipedia.org/wiki/Robots_exclusion_standard http://en.wikipedia.org/wiki/Robots_exclusion_standard