3 ms·
What's the appropriate user agents to disallow if you want to let search engines in, but not ML training crawlers?
by falcolas 4y ago
What's the appropriate user agents to disallow if you want to let search engines in, but not ML training crawlers?
- sharmi 4y agoYou will need to disallow all and enable specific search engine crawler ids Something akin to https://stackoverflow.com/questions/59092667/allow-only-googlebot-to-index-everything https://stackoverflow.com/questions/59092667/allow-only-goog...