5 ms·

Amazon's AI crawler is making my Git server unstable

33 points by bitbasher 2y ago

deleted 2y ago
[deleted]
JSTrading 2y ago
How much of a problem is it?
- xena 2y ago
  3Ti of egress and climbing, I'm in the hole financially and it's making my personal infra that relies on it unstable.
  - jsheard 2y ago
    Damn that's rude. At least you appear to be using Vultr, imagine if it was running on one of those newfangled cloud providers which mark bandwidth up by a few orders of magnitude...
    - xena 2y ago
      It's actually slightly worse. That vultr node is a reverse proxy over wireguard to my homelab.
      - bitbasher 2y ago
        Remove the gittea instance for now until it's sorted out? Respond to all git.* traffic with a 420 until it's sorted out.
alphan0n 2y ago
Have you verified that they are actually Amazon crawlers as outlined here: https://developer.amazon.com/amazonbot https://developer.amazon.com/amazonbot
- xena 2y ago
  Yes.
  - alphan0n 2y ago
    Can you share the list of offending ip’s, here or your website so we can use them in block lists? Also, there is an email to contact: amazonbot@amazon.com
    - xena 2y ago
      The website that you were pummeling is in the article: git.xeserv.us. I sent an email earlier today and have gotten no response. Right now your crawler bots are getting the bee movie script, so you may want to delete all the data that's being scraped from that domain. Unless you like jazz that is. It'd be a gesture of good faith to remunerate me for the egress fees your bot incurred, but I'm not gonna die on that hill.
      - alphan0n 2y ago
        Apologies, I’m not affiliated with Amazon in any way. I meant the Amazon ip addresses that are causing you trouble so I can preemptively block them.