2 ms·
If Google doesn't provide a way to say "please don't scrape my site", then it 100% unethical. We have robots.txt. If Google doesn't respect that, it's unethica
by raincole 3y ago
If Google doesn't provide a way to say "please don't scrape my site", then it 100% unethical.
We have robots.txt. If Google doesn't respect that, it's unethical. Don't you think so?
- pcthrowaway 3y agoMaybe it's time for an llm.txt Not that the people you want to respect that would
- raincole 3y agoThe tricky part is it's much more harder to prove that they didn't respect that.
- RobotToaster 3y agoDoes twitter's robots.txt forbid scraping? Judging by the fact it shows up in Google I'd assume not.