3 ms·
Their explanation for ignoring robots makes sense - they say they ignore it because their crawler only runs when a human enters a URL and archives it, they also
by jonathantf2 4y ago
Their explanation for ignoring robots makes sense - they say they ignore it because their crawler only runs when a human enters a URL and archives it, they also link to Google as this is what they do.