3 ms·it says it'll respect robots.txt if you don't want your page crawled (parsed? interpreted?)by fermuch 4y agoit says it'll respect robots.txt if you don't want your page crawled (parsed? interpreted?)