4 ms·
> Some of the bots identify themselves, but some don't. Either way, the respondents say that robots.txt directives – voluntary behavior guidelines that web publ
by rglover 1y ago
> Some of the bots identify themselves, but some don't. Either way, the respondents say that robots.txt directives – voluntary behavior guidelines that web publishers post for web crawlers – are not currently effective at controlling bot swarms.
Is anybody tracking the IP ranges of bots or anything similar that's reliable?
It seems like they're taking the "what are you gonna do about it" approach to this.
Edit: Yes [1]
[1] https://github.com/FabrizioCafolla/openai-crawlers-ip-ranges https://github.com/FabrizioCafolla/openai-crawlers-ip-ranges
- dbmikus 1y agoMany bots use residential IP proxy networks, so they come from the same IPs that humans use
- renegat0x0 1y agoI think that some of big tech already said that they don't respect robots in what is prohibited. It is highly probable that ordinary bots also does not respect robots.txt.
- zimpenfish 1y agoGPTBot certainly doesn't - I added a blanket disallow for it several months ago and in the last 5 days, it's done 22k requests (rate-limited to a max of 5 req/minute, all proxied to iocaine[0].) [0] https://iocaine.madhouse-project.org https://iocaine.madhouse-project.org