3 ms·
I believe most people respect robots.txt on your site. If they don't I don't think there is anything you can do about it but in general most companies do. At le
by PaulsWallet 4y ago
I believe most people respect robots.txt on your site. If they don't I don't think there is anything you can do about it but in general most companies do. At least most companies I've had personal experience with.
- falcolas 4y agoWhat's the appropriate user agents to disallow if you want to let search engines in, but not ML training crawlers?
- sharmi 4y agoYou will need to disallow all and enable specific search engine crawler ids Something akin to https://stackoverflow.com/questions/59092667/allow-only-googlebot-to-index-everything https://stackoverflow.com/questions/59092667/allow-only-goog...