4 ms·
I think for a big company they would get sued if they try to build an LLM just on a website's data without paying them anything for it. For a search engine they
by kats 3y ago
I think for a big company they would get sued if they try to build an LLM just on a website's data without paying them anything for it. For a search engine they can do that, but the search engine actually benefits the original sites. An LLM doesn't really send any money towards the original websites.
- realusername 3y agoThey already do that, the dataset is called Common Crawl...
- ryanbrunner 3y agoIf your robots.txt allows access to the user agent crawling (which Reddit does except for a very small denylist), and the crawler isn't making any attempt to bypass the site's security, they wouldn't have a leg to stand on legally.