3 ms·The Disallowed directories from the robots.txt files of the world's top websites2 points by gkop 10y ago