3 ms·
Nice. How do we force them to respect robots.txt?
by tecleandor 2y ago
Nice. How do we force them to respect robots.txt?
- szczepano 2y agothey respect robots.txt at least major ones like meta, claude, google, openai, based on my infra observations robots.txt is enough in 90%, 10% is just banning ip ranges for couple of days but those are no AI companies