5 ms·
That’s not going to be much of a roadblock for LLM trainers… they crawl the web just as well as they consume API output. I don’t think this will accomplish anyt
by kitsunesoba 3y ago
That’s not going to be much of a roadblock for LLM trainers… they crawl the web just as well as they consume API output. I don’t think this will accomplish anything but anger legitimate users.
- kats 3y agoI think for a big company they would get sued if they try to build an LLM just on a website's data without paying them anything for it. For a search engine they can do that, but the search engine actually benefits the original sites. An LLM doesn't really send any money towards the original websites.
- realusername 3y agoThey already do that, the dataset is called Common Crawl...
- ryanbrunner 3y agoIf your robots.txt allows access to the user agent crawling (which Reddit does except for a very small denylist), and the crawler isn't making any attempt to bypass the site's security, they wouldn't have a leg to stand on legally.
- toomuchtodo 3y agoOpenAI and other LLMs could offer browser extensions that scrape forums and send data off to them in the natural course of web browsing, no API required. No different than Recap for PACER with regards to mechanics. https://free.law/recap https://free.law/recap https://blog.law.cornell.edu/voxpop/2011/02/03/pacer-recap-and-the-movement-to-free-american-case-law/ https://blog.law.cornell.edu/voxpop/2011/02/03/pacer-recap-a...
- paradox460 3y agoSam Altman is on reddit's board. If he wants openai to have access to reddit data, he'll get it
- cratermoon 3y agoSam Altman's OpenAI already has Reddit's data. He is more interested in making sure nobody else can get it.
- throwaway5959 3y agoTOS + require the user to login in to browse Reddit pretty much solves that right?
- kitsunesoba 3y agoGoing login-only would be suicidal. A large chunk of Reddit’s traffic and probably the majority of new users come from search engines, which login walls lock out. It’s not like Twitter where most growth comes from existing users linking tweets and user profiles. TOS as far as I know is not legally enforceable, and even if it is there are people in countries that disregard such things who are training LLMs.
- ryanbrunner 3y agoTechnically they could allowlist Googlebot and deny everything else, but yeah I don't see them really doing that.
- Name_Chawps 3y agoIt's a legal barrier, not a practical one.