4 ms·
My personal VPS was recently inundated with 800GB/month of traffic from AI scraper bots. Upon a bit of investigation they were getting stuck in some deep git hi
by mappu 1y ago
My personal VPS was recently inundated with 800GB/month of traffic from AI scraper bots. Upon a bit of investigation they were getting stuck in some deep git history pages. I looked into Anubis and the like, but making carveouts for API endpoints seemed complicated.
Luckily the Gitea devs had recently implemented `REQUIRE_SIGNIN_VIEW = expensive` as a fix. It was minimally invasive for regular users, most pages can still be accessed without login, and it completely solved the AI bot problem, my traffic and load averages are back to normal.
Thank you Gitea devs for a great product, happy user for over a decade both personally and professionally.
- techknowlogick 1y agoThank YOU for using it :)
- rnhmjoj 1y agoI didn't know about this option, thanks. I had the same issue and solved it the hard way: I blackholed IP addresses from a bunch of ASNs (openai, microsoft, mistral).
- captn3m0 1y agoHad similar problems, but ended up migrating to rgit + gickup instead.