3 ms·It doesn't even respect general robots.txt exclusions, so... I doubt it.by duskwuff 3y agoIt doesn't even respect general robots.txt exclusions, so... I doubt it.