3 ms·
The difference is that, AFAIK, those bigger AI crawlers do respect robots.txt. Google even provides a way to opt-out of AI training without opting-out of search
by jsheard 2y ago
The difference is that, AFAIK, those bigger AI crawlers do respect robots.txt. Google even provides a way to opt-out of AI training without opting-out of search indexing.
- yazzku 2y agoAnd how much do you trust that shit? Has anyone set up a honeypot as an experiment?
- BXlnt2EachOther 2y agopossibly unpopular opinion, I trust the bigger companies more than small ones on stuff like this. It would be so much easier to not offer anything, rather than intentionally create a potemkin setting and risk the blowback that would occur if discovered. Hopefully this comment does not age poorly. full disclosure: worked there [edit: google] a while ago, not in search, not in AI.
- Arnt 2y agoYou can trust Google to do what it says, and yes I've seen Google obey robots.txt. You can't trust Google to do what you think is right.
- yazzku 2y agoNo, you can't: https://apnews.com/article/828aefab64d4411bac257a07c1af0ecb https://apnews.com/article/828aefab64d4411bac257a07c1af0ecb
- Arnt 2y agoI'm a bit in a hurry, don't have time for close reading. Does that article say some Google apps (notably Maps) store locations on your device even if you have configured them to not store it in your Google account? I may miss something, don't have time to read between the lines today.