2 ms·

Global tarpit is the solution. It makes sense anyway even without taking AI crawlers into account. Back when I had to implement that, I went the semi manual rou

Global tarpit is the solution. It makes sense anyway even without taking AI crawlers into account. Back when I had to implement that, I went the semi manual route - parse the access log and any IP address averaging more than X hits a second on /api gets a -j TARPIT with iptables [1]. Not sure how to implement it in the cloud though, never had the need for that there yet. [1] https://gist.github.com/flaviovs/103a0dbf62c67ff371ff75fc62fdded3 https://gist.github.com/flaviovs/103a0dbf62c67ff371ff75fc62f...

jks 2y ago
One such tarpit (Nepenthes) was just recently mentioned on Hacker News: https://news.ycombinator.com/item?id=42725147 https://news.ycombinator.com/item?id=42725147 Their site is down at the moment, but luckily they haven't stopped Wayback Machine from crawling it: https://web.archive.org/web/20250117030633/https://zadzmo.org/code/nepenthes/ https://web.archive.org/web/20250117030633/https://zadzmo.or...
- kazinator 2y ago
  How do you know their site is down? You probably just hit their tarpit. :)
- marcus0x62 2y ago
  Quixotic[0] (my content obfuscator) includes a tarpit component, but for something like this, I think the main quixotic tool would be better - you run it against your content once, and it generates a pre-obfuscated version of it. It takes a lot less of your resources to serve than dynamically generating the tarpit links and content. 0 - https://marcusb.org/hacks/quixotic.html https://marcusb.org/hacks/quixotic.html
bwfan123 2y ago
i would think public outcry by influencers on social media (such as this thread) is a better deterrent, and also establishes a public datapoint and exhibit for future reference.. as it is hard to scale the tarpit.
idlewords 2y ago
This doesn't work with the kind of highly distributed crawling that is the problem now.
seethenerdz 2y ago
Don't we have intellectual property law for this tho?

jks 2y ago

One such tarpit (Nepenthes) was just recently mentioned on Hacker News: https://news.ycombinator.com/item?id=42725147 https://news.ycombinator.com/item?id=42725147 Their site is down at the moment, but luckily they haven't stopped Wayback Machine from crawling it: https://web.archive.org/web/20250117030633/https://zadzmo.org/code/nepenthes/ https://web.archive.org/web/20250117030633/https://zadzmo.or...

kazinator 2y ago
How do you know their site is down? You probably just hit their tarpit. :)
marcus0x62 2y ago
Quixotic[0] (my content obfuscator) includes a tarpit component, but for something like this, I think the main quixotic tool would be better - you run it against your content once, and it generates a pre-obfuscated version of it. It takes a lot less of your resources to serve than dynamically generating the tarpit links and content. 0 - https://marcusb.org/hacks/quixotic.html https://marcusb.org/hacks/quixotic.html

bwfan123 2y ago

i would think public outcry by influencers on social media (such as this thread) is a better deterrent, and also establishes a public datapoint and exhibit for future reference.. as it is hard to scale the tarpit.

idlewords 2y ago

This doesn't work with the kind of highly distributed crawling that is the problem now.

seethenerdz 2y ago

Don't we have intellectual property law for this tho?