5 ms·
Amazon's AI crawler is making my Git server unstable
- osdotsystem 2y agoI feel for you. Same for OpenAI (https://www.linkedin.com/posts/appinv_openai-is-a-felon-company-with-pirate-altman-activity-7286300344869949440-WI8I https://www.linkedin.com/posts/appinv_openai-is-a-felon-comp...). A friend told me Ai bots even ignored the robots.tx ...
- Arnt 2y agoI've heard that too. Your posting inspired me to check. I see daily accesses from claudebot and gtpbot to /robots.txt on two sites. Also from something called amazonbot, which may be another AI bot?
- osdotsystem 2y agoMe i dropped a bunch of dont allows for different bots, as many as i could find.
- Arnt 2y agoI'm curious — if that's your attitude, why not all bots? You can say "User-Agent: foo\nDisallow:\n\nUser-Agent: *\nDisallow: /" to allow only named bots and reject them by default.
- fragmede 2y ago> What else do I need to do? I mean, not sending 418 and sending 429 or 403 or anything more useful would be on my list of things to try. Might double check what 418 is while you're trying to figure out why sending that doesn't seem to do anything.
- xena 2y agoChanging the status code didn't help. I eventually just withdrew the server from the internet.
- LinuxBender 2y agoNginx has an assortment of options to rate limit people/bots. [1] For the 10% not using a user-agent generic rate limits could be applied based on volume limit_rate_after. One would have to get creative with nginx maps. Another option would be to calculate patterns in your access logs and just blackhole or ipset reject IP addresses or networks that are abusive accepting some of them may be abusive humans vs abusive bots. [1] - https://serverfault.com/questions/639671/nginx-how-to-limit-request-rate-based-on-user-agent https://serverfault.com/questions/639671/nginx-how-to-limit-...
- deleted 2y ago[deleted]