3 ms·

You’re so right. I have a public-facing Forgejo server. Before configuring Anubis, scrapers were sending it about 600K requests per day. Copying and pasting fro

You’re so right. I have a public-facing Forgejo server. Before configuring Anubis, scrapers were sending it about 600K requests per day. Copying and pasting from my blog post about it: * For every Git commit, fetch the version of every file in the repository at that commit. * See git blame for every file at every commit. * Attempt to download the archive of each repo at every commit. * Run every possible pull request search filter combination. * Run every possible issue search filter combination. * Fetch each of those URLs at random from some residential IP in Brazil that had not ever accessed my server before. Afterward, it dropped to several hundred. Expect anti-attack features to keep getting stranger and more visible as scraper get still more aggressive.

andai 3mo ago
That's crazy. So 600,000 someones are absolutely starved for data. Or one someone with too much money and too little sense misplaced a decimal point in their ScraPy setup?
- kstrauser 3mo ago
  I think it was the latter. And the cruel irony is that these are FOSS Git repos I'm publicly sharing. I'd've been fine with them cloning the repo and analyzing away to their heart's content. That's not the way their scraper's wired, though.
grep_name 3mo ago
What do you use to monitor this? I don't really keep a close enough eye on my services to know what the traffic is doing (and haven't had any issues) but maybe I should start
- kstrauser 3mo ago
  I just tailed the web logs for a bit and saw that it was wild. For fun, I fed 10 minutes of logs into an AI and it picked up a lot of signal I didn't catch at first glance, like clients claiming to be MSIE 7 on Android 3 and such. I added some reject rules to the webserver in front of Forgejo but that only made a dent in the traffic, alas.
lloydatkinson 3mo ago
This matches my experience of running a public mediawiki server. The bots (mostly Facebook/Meta) will for every single change fetch every single page again, over and over. I can’t tell if it’s incompetence or malice.

andai 3mo ago

That's crazy. So 600,000 someones are absolutely starved for data. Or one someone with too much money and too little sense misplaced a decimal point in their ScraPy setup?

kstrauser 3mo ago
I think it was the latter. And the cruel irony is that these are FOSS Git repos I'm publicly sharing. I'd've been fine with them cloning the repo and analyzing away to their heart's content. That's not the way their scraper's wired, though.

grep_name 3mo ago

What do you use to monitor this? I don't really keep a close enough eye on my services to know what the traffic is doing (and haven't had any issues) but maybe I should start

kstrauser 3mo ago
I just tailed the web logs for a bit and saw that it was wild. For fun, I fed 10 minutes of logs into an AI and it picked up a lot of signal I didn't catch at first glance, like clients claiming to be MSIE 7 on Android 3 and such. I added some reject rules to the webserver in front of Forgejo but that only made a dent in the traffic, alas.

lloydatkinson 3mo ago

This matches my experience of running a public mediawiki server. The bots (mostly Facebook/Meta) will for every single change fetch every single page again, over and over. I can’t tell if it’s incompetence or malice.