3 ms·

I had the same issue when I first put up my gitea instance. The bots found the domain through cert registration in minutes, before there were any backlinks. GPT

by rozab 8mo ago

I had the same issue when I first put up my gitea instance. The bots found the domain through cert registration in minutes, before there were any backlinks. GPTbot, ClaudeBot, PerplexityBot, and others. I added a robots.txt with explicit UAs for known scrapers (they seem to ignore wildcards), and after a few days the traffic died down completely and I've had no problem since. Git frontends are basically a tarpit so are uniquely vulnerable to this, but I wonder if these folks actually tried a good robots.txt? I know it's wrong that they ignore wildcards, but it does seem to solve the issue

bob1029 8mo ago
> I wonder if these folks actually tried a good robots.txt? I suspect that some of these folks are not interested in a proper solution. Being able to vaguely claim that the AI boogeyman is oppressing us has turned into quite the pastime.
- embedding-shape 8mo ago
  > Being able to vaguely claim that the AI boogeyman is oppressing us has turned into quite the pastime. FWIW, you're literally in a comment thread where GP (me!) says "don't understand what the big issue is"...
trillic 8mo ago
I will second a good robots.txt. Just checked my metrics and < 100 requests total to my git instance in the last 48 hours. Completely public, most repos are behind a login but there are a couple that are public and linked.
stefanka 8mo ago
Where does one find a good robots.txt? Are there any well maintained out there?
- skrtskrt 8mo ago
  Cloudflare actually has this as a free tier feature so even if you don't want to use it for your site you can just setup a throwaway domain on Cloudflare and periodically copy the robots.txt they generate from your scraper allow/block preferences, since they'll be keeping up to date with all the latest.
- dmit 8mo ago
  https://github.com/ai-robots-txt/ai.robots.txt https://github.com/ai-robots-txt/ai.robots.txt

by rozab 8mo ago

bob1029 8mo ago

> I wonder if these folks actually tried a good robots.txt? I suspect that some of these folks are not interested in a proper solution. Being able to vaguely claim that the AI boogeyman is oppressing us has turned into quite the pastime.

embedding-shape 8mo ago
> Being able to vaguely claim that the AI boogeyman is oppressing us has turned into quite the pastime. FWIW, you're literally in a comment thread where GP (me!) says "don't understand what the big issue is"...

trillic 8mo ago

I will second a good robots.txt. Just checked my metrics and < 100 requests total to my git instance in the last 48 hours. Completely public, most repos are behind a login but there are a couple that are public and linked.

stefanka 8mo ago

Where does one find a good robots.txt? Are there any well maintained out there?

skrtskrt 8mo ago
Cloudflare actually has this as a free tier feature so even if you don't want to use it for your site you can just setup a throwaway domain on Cloudflare and periodically copy the robots.txt they generate from your scraper allow/block preferences, since they'll be keeping up to date with all the latest.
dmit 8mo ago
https://github.com/ai-robots-txt/ai.robots.txt https://github.com/ai-robots-txt/ai.robots.txt