2 ms·
Your best hope is to add a section to this page: https://www.robotstxt.org/robotstxt.html https://www.robotstxt.org/robotstxt.html The dev community should set
by tiborsaas 4y ago
Your best hope is to add a section to this page: https://www.robotstxt.org/robotstxt.html https://www.robotstxt.org/robotstxt.html
The dev community should settle on a string that would mean that bots which are scraping data for ML projects should stop indexing.
> Is it ethical that OpenAI, MS, Google, and others monetize their AI models while using the web data without asking for permission from the authors?
They already made fortunes and empires based on indexing web data so there's no hope there will be a law to prohibit a single, much more narrower use-case for web crawling.