3 ms·

Something I want to ask before: I've noticed that some people have made a connection between the decision to effectively shut down Reddit's API and the increas

Something I want to ask before: I've noticed that some people have made a connection between the decision to effectively shut down Reddit's API and the increasing popularity of LLM/ChatGPT, and it does seem plausible. What's interesting is that Twitter also closed down their API access merely months ago. Are these two occurrences related, or was Twitter's action also a response to ChatGPT/LLM? (Also to be totally fair to them, Twitter and Reddit used to have the most comprehensive free API accesses which I enjoy. Instagram/facebook etc. are always a nightmare to work with in term of scraping.)

hn_throwaway_99 3y ago
That weird "conspiracy theory" doesn't make any sense. LLMs are trained off a static dump of example data, and it's trivially easy to build a webscraper of Reddit's historical content.
- buildbot 3y ago
  It's trivially easy to build a scraper for the new content too. And I think people really over estimate the relative amount of data reddit has compared to the literal rest of the internet. Like sure, it's nicely categorized and ranked (somewhat), but leaving reddit out of a corpus probably won't drastically effect your final model.
Accujack 3y ago
>some people have made a connection between the decision to effectively shut down Reddit's API and the increasing popularity of LLM/ChatGPT Those people really don't have an idea how either of those work, nor how cause and effect work. See also the graph showing that the decrease in the number of pirates in the world's oceans has contributed to climate change because "pirates are cool".
SirSourdough 3y ago
I believe this is more than speculation. Pretty sure that someone from Reddit basically said as much in one of the transcripts or conversations shared by Apollo’s dev. Reddit feels they are getting burned by letting their data be used to train LLMs for free. That said, they were also letting people exceed their API rate limits by massive amounts, so hard for me to put the blame anywhere but with Reddit.
lemming 3y ago
Here's an interesting theory about that. I can't claim any credit for it, it's from Jim Rutt, I heard him on the Clearer Thinking podcast (https://podcast.clearerthinking.org/episode/160/jim-rutt-large-language-models-deep-peace-and-the-meaning-crisis/ https://podcast.clearerthinking.org/episode/160/jim-rutt-lar...). He basically argues that LLMs are going to mean that vast quantities of the content on the internet are going to become automatically-generated drivel which no-one wants to actually read. I think this is pretty plausible. So in order to consume information from the internet, it will basically be a requirement to use an AI tool which, for a given query that you might have, will go out and retrieve all the information on it from many different sources, and return them, appropriately listed and summarised. Such a tool would probably want API access to the likes of Reddit, but would also make the main value prop of the way social media works currently (getting ads in front of eyeballs) useless. So end users will probably end up having to pay for their API use so that their tools can access these sites, and as a pleasant side effect they'll see no ads. So perhaps we'll end up at the promised utopia where users pay a little bit for the content that they like, and don't see ads along the way, as a result of AI. I have no idea how likely this is, but it sounds pretty good to me if it does work out like that.
- b800h 3y ago
  And then they'll put ads in the LLM chat stream. "Using texts on neurolinguistic programming and mentalism, find subliminal ways of promoting {product}".
boomboomsubban 3y ago
The CEO is explicitly saying this change has to do with the ChatGPT's and such. >But with artificial intelligence-powered large language models like Microsoft-backed ChatGPT and Google's Bard, a massive corpus of conversations is being hoovered up. And in return, Reddit receives very little, he said. https://www.npr.org/2023/06/15/1182457366/reddit-ceo-steve-huffman-its-time-we-grow-up-and-behave-like-an-adult-company https://www.npr.org/2023/06/15/1182457366/reddit-ceo-steve-h... And this isn't the first time they've said this.
ttepasse 3y ago
They could simply discriminate against LLMs via their APIs terms of service. That wouldn’t get bad actors, those would simply scrape. But that would be the case with the very expensive API.

hn_throwaway_99 3y ago

That weird "conspiracy theory" doesn't make any sense. LLMs are trained off a static dump of example data, and it's trivially easy to build a webscraper of Reddit's historical content.

buildbot 3y ago
It's trivially easy to build a scraper for the new content too. And I think people really over estimate the relative amount of data reddit has compared to the literal rest of the internet. Like sure, it's nicely categorized and ranked (somewhat), but leaving reddit out of a corpus probably won't drastically effect your final model.

Accujack 3y ago

>some people have made a connection between the decision to effectively shut down Reddit's API and the increasing popularity of LLM/ChatGPT Those people really don't have an idea how either of those work, nor how cause and effect work. See also the graph showing that the decrease in the number of pirates in the world's oceans has contributed to climate change because "pirates are cool".

SirSourdough 3y ago

I believe this is more than speculation. Pretty sure that someone from Reddit basically said as much in one of the transcripts or conversations shared by Apollo’s dev. Reddit feels they are getting burned by letting their data be used to train LLMs for free. That said, they were also letting people exceed their API rate limits by massive amounts, so hard for me to put the blame anywhere but with Reddit.

lemming 3y ago

Here's an interesting theory about that. I can't claim any credit for it, it's from Jim Rutt, I heard him on the Clearer Thinking podcast (https://podcast.clearerthinking.org/episode/160/jim-rutt-large-language-models-deep-peace-and-the-meaning-crisis/ https://podcast.clearerthinking.org/episode/160/jim-rutt-lar...). He basically argues that LLMs are going to mean that vast quantities of the content on the internet are going to become automatically-generated drivel which no-one wants to actually read. I think this is pretty plausible. So in order to consume information from the internet, it will basically be a requirement to use an AI tool which, for a given query that you might have, will go out and retrieve all the information on it from many different sources, and return them, appropriately listed and summarised. Such a tool would probably want API access to the likes of Reddit, but would also make the main value prop of the way social media works currently (getting ads in front of eyeballs) useless. So end users will probably end up having to pay for their API use so that their tools can access these sites, and as a pleasant side effect they'll see no ads. So perhaps we'll end up at the promised utopia where users pay a little bit for the content that they like, and don't see ads along the way, as a result of AI. I have no idea how likely this is, but it sounds pretty good to me if it does work out like that.

b800h 3y ago
And then they'll put ads in the LLM chat stream. "Using texts on neurolinguistic programming and mentalism, find subliminal ways of promoting {product}".

boomboomsubban 3y ago

The CEO is explicitly saying this change has to do with the ChatGPT's and such. >But with artificial intelligence-powered large language models like Microsoft-backed ChatGPT and Google's Bard, a massive corpus of conversations is being hoovered up. And in return, Reddit receives very little, he said. https://www.npr.org/2023/06/15/1182457366/reddit-ceo-steve-huffman-its-time-we-grow-up-and-behave-like-an-adult-company https://www.npr.org/2023/06/15/1182457366/reddit-ceo-steve-h... And this isn't the first time they've said this.

ttepasse 3y ago

They could simply discriminate against LLMs via their APIs terms of service. That wouldn’t get bad actors, those would simply scrape. But that would be the case with the very expensive API.