3 ms·

I'm sure their crawler can handle a zip bomb. Plus it might interpret that as "this site doesn't have a robots.txt" and start scraping that OP is trying to prev

by delecti 8mo ago

I'm sure their crawler can handle a zip bomb. Plus it might interpret that as "this site doesn't have a robots.txt" and start scraping that OP is trying to prevent with their current robots.txt.

1e1a 8mo ago
Could allow only the path to the zip bomb for this user agent.
- FartyMcFarter 8mo ago
  That will work once at most and then quickly get fixed.
  - esseph 8mo ago
    Are you so sure? :)
  - xp84 8mo ago
    Yeah it seems like this team takes a really tough stance on obvious bugs
marginalia_nu 8mo ago
Pretty sure every crawler can. You kinda have to go out of your way not to, given how the gzread API looks. https://refspecs.linuxbase.org/LSB_3.0.0/LSB-Core-generic/LSB-Core-generic/zlib-gzread-1.html https://refspecs.linuxbase.org/LSB_3.0.0/LSB-Core-generic/LS...