2 ms·
If you're not interested in letting your blog posts be used to train AI, remember to update your robots.txt regularly as AI crawlers come up. This is assuming t
by prophesi 2y ago
If you're not interested in letting your blog posts be used to train AI, remember to update your robots.txt regularly as AI crawlers come up. This is assuming they respect your robots.txt, and don't change their user-agent. There's a service for this called Dark Visitors[0], though I wish there was a FOSS project to track AI user-agents.
[0] https://darkvisitors.com/ https://darkvisitors.com/