4 ms·

I do a similar thing with web crawlers that do not respect the robots.txt https://github.com/cl-test-grid/cl-test-grid/blob/873b2fa9788f0417d7679c2c778c30011fb

I do a similar thing with web crawlers that do not respect the robots.txt https://github.com/cl-test-grid/cl-test-grid/blob/873b2fa9788f0417d7679c2c778c30011fbadb07/gae-blobstore/server/src/cltestgrid/GetBlob2.java#L30 https://github.com/cl-test-grid/cl-test-grid/blob/873b2fa978... I don't know if this snippet is really effective, can be improved a little, especially that I noticed a couple of new crawlers that ignore `User-agent: * Disallow: /path` in robots.txt, and do not fix that even after reported.

sodimel 4y ago
That's pretty neat! I should set up something similar for my domains that keep being spammed.
amiga-workbench 4y ago
I wonder if you could abuse gzip compression on responses to send a zip bomb back to them.
- Avamander 4y ago
  You can, but most bots do timeout. If you get a lot of bad bots that are vulnerable, then you'll probably waste a lot of resources on those connections.
- winddude 4y ago
  Yes you can. There ar also ways to protect the crawler server from crashing.
- dawnerd 4y ago
  I did just that for a while with a spare server I had. I set it up to literally only respond to bad bots. I know the crawlers don't care but it amused me at least. I tried to also keep redirecting slowly before it could time out. There was one bot that seemed to create a new instance each redirect so I could keep it in a loop for essentially ever. Just about every other bot only followed a few redirects before giving up. Fun times.
gigel82 4y ago
I'll look into implementing this, nice tip :) archive.org is the worst offender for me; not only do they ignore robots.txt, there is absolutely no way to get something removed once they archived it (despite the data including accidentally leaked PII for example - which can cause actual harm to someone).
- ShakataGaNai 4y ago
  Just send them a DMCA request, that's their takedown mechanism. Is it a good one? No, but that's how they do it. You see it posted about all over in their forums. ex: https://archive.org/post/1022869/site-removal-request https://archive.org/post/1022869/site-removal-request
  - gigel82 4y ago
    They ignored all emails sent to info@archive.org from the actual domain in question which I owned, with a link to a URL on the domain asking them to remove it. Don't know what the DMCA process is, I presume it involves lawyers and such.
- Analemma_ 4y ago
  I want archive.org to ignore robots.txt and make it as difficult as possible to remove pages from it; it would be a broken archive tool if this were not the case.
  - anonymousiam 4y ago
    Sometimes there's a legitimate need to remove archived content. Back in the early days of running an ISP, I had some users hide links to copyrighted content with "map name" in a hosted image. Eventually the search engines began indexing these links and my net traffic exploded. I was not happy.
  - gigel82 4y ago
    I hope you never accidentally publish private information anywhere on the web.

sodimel 4y ago

That's pretty neat! I should set up something similar for my domains that keep being spammed.

amiga-workbench 4y ago

I wonder if you could abuse gzip compression on responses to send a zip bomb back to them.

Avamander 4y ago
You can, but most bots do timeout. If you get a lot of bad bots that are vulnerable, then you'll probably waste a lot of resources on those connections.
winddude 4y ago
Yes you can. There ar also ways to protect the crawler server from crashing.
dawnerd 4y ago
I did just that for a while with a spare server I had. I set it up to literally only respond to bad bots. I know the crawlers don't care but it amused me at least. I tried to also keep redirecting slowly before it could time out. There was one bot that seemed to create a new instance each redirect so I could keep it in a loop for essentially ever. Just about every other bot only followed a few redirects before giving up. Fun times.

gigel82 4y ago

I'll look into implementing this, nice tip :) archive.org is the worst offender for me; not only do they ignore robots.txt, there is absolutely no way to get something removed once they archived it (despite the data including accidentally leaked PII for example - which can cause actual harm to someone).

ShakataGaNai 4y ago
Just send them a DMCA request, that's their takedown mechanism. Is it a good one? No, but that's how they do it. You see it posted about all over in their forums. ex: https://archive.org/post/1022869/site-removal-request https://archive.org/post/1022869/site-removal-request
- gigel82 4y ago
  They ignored all emails sent to info@archive.org from the actual domain in question which I owned, with a link to a URL on the domain asking them to remove it. Don't know what the DMCA process is, I presume it involves lawyers and such.
Analemma_ 4y ago
I want archive.org to ignore robots.txt and make it as difficult as possible to remove pages from it; it would be a broken archive tool if this were not the case.
- anonymousiam 4y ago
  Sometimes there's a legitimate need to remove archived content. Back in the early days of running an ISP, I had some users hide links to copyrighted content with "map name" in a hosted image. Eventually the search engines began indexing these links and my net traffic exploded. I was not happy.
- gigel82 4y ago
  I hope you never accidentally publish private information anywhere on the web.