4 ms·

Have you considered serving a zip bomb to this user agent?

• delecti 7mo ago
  I'm sure their crawler can handle a zip bomb. Plus it might interpret that as "this site doesn't have a robots.txt" and start scraping that OP is trying to prevent with their current robots.txt.
  • 1e1a 7mo ago
    Could allow only the path to the zip bomb for this user agent.
    • FartyMcFarter 7mo ago
      That will work once at most and then quickly get fixed.
      • esseph 7mo ago
        Are you so sure? :)
      • xp84 7mo ago
        Yeah it seems like this team takes a really tough stance on obvious bugs
  • marginalia_nu 7mo ago
    Pretty sure every crawler can. You kinda have to go out of your way not to, given how the gzread API looks. https://refspecs.linuxbase.org/LSB_3.0.0/LSB-Core-generic/LSB-Core-generic/zlib-gzread-1.html https://refspecs.linuxbase.org/LSB_3.0.0/LSB-Core-generic/LS...