3 ms·
I think that some of big tech already said that they don't respect robots in what is prohibited. It is highly probable that ordinary bots also does not respect
by renegat0x0 1y ago
I think that some of big tech already said that they don't respect robots in what is prohibited. It is highly probable that ordinary bots also does not respect robots.txt.
- zimpenfish 1y agoGPTBot certainly doesn't - I added a blanket disallow for it several months ago and in the last 5 days, it's done 22k requests (rate-limited to a max of 5 req/minute, all proxied to iocaine[0].) [0] https://iocaine.madhouse-project.org https://iocaine.madhouse-project.org