2 ms·
I manually blacklisted .gov from being crawled on my side. felt like it wasn't worth the potential trouble
by PatchRequest 5mo ago
I manually blacklisted .gov from being crawled on my side. felt like it wasn't worth the potential trouble
- marysol5 5mo agoMeanwhile you crawl anybody else? .gov isn't magically any more able to take you to court than a private org