3 ms·

I'm having lots of connections every day from Singapor. It's now the main country... despite the whole website being French-only. AI crawlers, for sure. Thanks

by maelito 10mo ago

I'm having lots of connections every day from Singapor. It's now the main country... despite the whole website being French-only. AI crawlers, for sure. Thanks for this tip.

arjie 10mo ago
Amazonbot does this despite my efforts in robots.txt to help it out. I look at all the Singapore requests and they’re Amazonbot trying to get various variants of the Special:RecentChanges page. You’re wasting your time, Amazonbot. I’m trying to help you.
- reconnecting 10mo ago
  Did you check IP address of this UA?
  - arjie 10mo ago
    Yeah, a while ago, they're all Singapore reporting Amazonbot. Here is an example request: "GET /w/A_Wedding_at_City_Hall HTTP/1.1" 200 9677 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36" The actual IP is in X-Forwarded-For and I didn't keep that.
    - reconnecting 10mo ago
      This is just a spoofed UA, because Amazon has a publicly available range of their bots and they are not in Singapore.
      - arjie 10mo ago
        That makes sense. I wonder why Amazonbot specifically as a target to spoof. I hoped to get them not stuck using a robots.txt but they refuse to obey it and keep hitting that page with various params. No problem for me, but they are going nowhere.
        reconnecting 10mo ago
        Perhaps result of A/B testing, because web masters less frequently block them.
input_sh 10mo ago
Fun fact: you don't get rid of them even when you put a captcha on all visitors from Singapore. I still see a spike in traffic that perfectly matches the spike in served captchas, but this time it's geographically distributed between places like Iraq, Bangladesh and Brazil. Hopefully it at least costs them a little bit more.
- reconnecting 10mo ago
  Usually, there are multiple layers of different counter-protection measures. If you block by country, they shift to different IP ranges, if you block by IP, they might use a new IP for every request, and escalate further depending on the bot owner and your actions.
sunaookami 10mo ago
Yeah same for my Gitea instance. These were all ByteDance and Tencent ASNs from some AWS-equivalent. Blocked the whole subnet belonging to them in my server's ufw and haven't had any problems since then. Same for Vultr and Google Cloud.

by maelito 10mo ago

I'm having lots of connections every day from Singapor. It's now the main country... despite the whole website being French-only. AI crawlers, for sure. Thanks for this tip.

arjie 10mo ago

Amazonbot does this despite my efforts in robots.txt to help it out. I look at all the Singapore requests and they’re Amazonbot trying to get various variants of the Special:RecentChanges page. You’re wasting your time, Amazonbot. I’m trying to help you.

reconnecting 10mo ago
Did you check IP address of this UA?
- arjie 10mo ago
  Yeah, a while ago, they're all Singapore reporting Amazonbot. Here is an example request: "GET /w/A_Wedding_at_City_Hall HTTP/1.1" 200 9677 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36" The actual IP is in X-Forwarded-For and I didn't keep that.
  - reconnecting 10mo ago
    This is just a spoofed UA, because Amazon has a publicly available range of their bots and they are not in Singapore.
    - arjie 10mo ago
      That makes sense. I wonder why Amazonbot specifically as a target to spoof. I hoped to get them not stuck using a robots.txt but they refuse to obey it and keep hitting that page with various params. No problem for me, but they are going nowhere.
      - reconnecting 10mo ago
        Perhaps result of A/B testing, because web masters less frequently block them.

input_sh 10mo ago

Fun fact: you don't get rid of them even when you put a captcha on all visitors from Singapore. I still see a spike in traffic that perfectly matches the spike in served captchas, but this time it's geographically distributed between places like Iraq, Bangladesh and Brazil. Hopefully it at least costs them a little bit more.

reconnecting 10mo ago
Usually, there are multiple layers of different counter-protection measures. If you block by country, they shift to different IP ranges, if you block by IP, they might use a new IP for every request, and escalate further depending on the bot owner and your actions.

sunaookami 10mo ago

Yeah same for my Gitea instance. These were all ByteDance and Tencent ASNs from some AWS-equivalent. Blocked the whole subnet belonging to them in my server's ufw and haven't had any problems since then. Same for Vultr and Google Cloud.