4 ms·

I manually blacklisted .gov from being crawled on my side. felt like it wasn't worth the potential trouble

• marysol5 3mo ago
  Meanwhile you crawl anybody else? .gov isn't magically any more able to take you to court than a private org