3 ms·

Because technology aside, its a new group of well funded companies who act as parasites on this already struggling industry. I know HN (and probably SV at larg

Because technology aside, its a new group of well funded companies who act as parasites on this already struggling industry. I know HN (and probably SV at large) has an anti-journalist bent, but ultimately by draining the individuals and companies that do the reporting, they wont have new material to scrape in the future. (Disclosure - I'm a former journalist building a way for the news industry to fight back. https://letsgo.forth.news https://letsgo.forth.news)

mrkramer 2y ago
News industry is struggling but scraping public web content is legal as far as it is not commercially used....at least that's my layman understanding (but Perplexity is using it commercially). The solution for news industry with regards to LLMs is licensing your news and content to LLMs products because they are going to keep scraping it anyways.
- jaredwiener 2y ago
  "You might as well make a deal with the people stealing your content because they are going to keep stealing it anyway" is not a great option. Part of licensing is the implicit understanding that if the parties do not come to a deal, they cannot use the licensed content. It seems from this article that Perplexity is not operating in good faith, ignoring robots.txt, using unpublished IP addresses, and then lying about the amount of referral traffic generated. Not to mention, whether or not its legal (and im not a lawyer) doesn't seem to be the issue. It's being so shortsighted that you build a business that steals a product from an unwitting third party, and when that third party is drained of resources, where will you pull from?
  - mrkramer 2y ago
    >ignoring robots.txt Robots.txt is not legally binding, it is a good will among web stakeholders to respect it. But if you are repeatedly breaking it, you will have an army of web masters against you. And as far as I remember, big news outlets are already suing LLMs companies or are already licensing their content to them.
    - jaredwiener 2y ago
      Yes to all of your points, but I think you are missing my distinction between "potentially legal" and "not being terrible." And yes, of course they are suing the LLM companies. Because they shouldn't be stealing their stuff, regardless of robots.txt.
segasaturn 2y ago
Well, as of the last 10 years the journalism industry has just been a lazy tweet regurgitation factory. The pipeline looks something like Tweets -> Journalists -> LLMs. So maybe it would be best if we cut out the middlemen?
- jaredwiener 2y ago
  If that were true then why not just index the tweets?
- claudiulodro 2y ago
  To be blunt, this seems like a lazy regurgitation of a tweet. There are lots of organizations out there doing really good work. Look at a couple of the Pulitzer winners from the last couple years for example: - Mississippi Today spent more than 5 years investigating and reporting on the welfare agency and uncovered huge corruption going all the way to the governor's office: https://mississippitoday.org/2023/05/08/pulitzer-prize-winning-backchannel/ https://mississippitoday.org/2023/05/08/pulitzer-prize-winni... - Lookout Santa Cruz was on the ground reporting during catastrophic flooding this year, including setting up a text message service to get information to people because the internet was unreliable: https://lookout.co/lookout-santa-cruz-wins-pulitzer-prize-for-breaking-news-reporting/ https://lookout.co/lookout-santa-cruz-wins-pulitzer-prize-fo... This is the stuff you lose without journalism.
- lancesells 2y ago
  LLMs are the middlemen. Search engines are the middlemen. Social networks are the middlemen. Go to the NY Times or LA Times and tell me it's based on Tweets. If you're talking about a place like The Verge which is a tech blog with journalism, they have articles based on tweets but also plenty that are legitimate, researched articles.
  - segasaturn 2y ago
    If that was true then what do journalists have to be worried about? Why would they need to "fight back" against the LLMs as the above poster said - as you know, ChatGPT can't do original research or investigations on its own. The truth is that the bulk of what is called "journalism" today isn't on-the-ground gumshoes reporting, it's rewording tweets/press releases/PR statements and maybe a couple of phone calls mixed in. A good rule of thumb is that anything that can be replaced by a current-gen AI wasn't worth much to begin with.

mrkramer 2y ago

News industry is struggling but scraping public web content is legal as far as it is not commercially used....at least that's my layman understanding (but Perplexity is using it commercially). The solution for news industry with regards to LLMs is licensing your news and content to LLMs products because they are going to keep scraping it anyways.

jaredwiener 2y ago
"You might as well make a deal with the people stealing your content because they are going to keep stealing it anyway" is not a great option. Part of licensing is the implicit understanding that if the parties do not come to a deal, they cannot use the licensed content. It seems from this article that Perplexity is not operating in good faith, ignoring robots.txt, using unpublished IP addresses, and then lying about the amount of referral traffic generated. Not to mention, whether or not its legal (and im not a lawyer) doesn't seem to be the issue. It's being so shortsighted that you build a business that steals a product from an unwitting third party, and when that third party is drained of resources, where will you pull from?
- mrkramer 2y ago
  >ignoring robots.txt Robots.txt is not legally binding, it is a good will among web stakeholders to respect it. But if you are repeatedly breaking it, you will have an army of web masters against you. And as far as I remember, big news outlets are already suing LLMs companies or are already licensing their content to them.
  - jaredwiener 2y ago
    Yes to all of your points, but I think you are missing my distinction between "potentially legal" and "not being terrible." And yes, of course they are suing the LLM companies. Because they shouldn't be stealing their stuff, regardless of robots.txt.

segasaturn 2y ago

Well, as of the last 10 years the journalism industry has just been a lazy tweet regurgitation factory. The pipeline looks something like Tweets -> Journalists -> LLMs. So maybe it would be best if we cut out the middlemen?

jaredwiener 2y ago
If that were true then why not just index the tweets?
claudiulodro 2y ago
To be blunt, this seems like a lazy regurgitation of a tweet. There are lots of organizations out there doing really good work. Look at a couple of the Pulitzer winners from the last couple years for example: - Mississippi Today spent more than 5 years investigating and reporting on the welfare agency and uncovered huge corruption going all the way to the governor's office: https://mississippitoday.org/2023/05/08/pulitzer-prize-winning-backchannel/ https://mississippitoday.org/2023/05/08/pulitzer-prize-winni... - Lookout Santa Cruz was on the ground reporting during catastrophic flooding this year, including setting up a text message service to get information to people because the internet was unreliable: https://lookout.co/lookout-santa-cruz-wins-pulitzer-prize-for-breaking-news-reporting/ https://lookout.co/lookout-santa-cruz-wins-pulitzer-prize-fo... This is the stuff you lose without journalism.
lancesells 2y ago
LLMs are the middlemen. Search engines are the middlemen. Social networks are the middlemen. Go to the NY Times or LA Times and tell me it's based on Tweets. If you're talking about a place like The Verge which is a tech blog with journalism, they have articles based on tweets but also plenty that are legitimate, researched articles.
- segasaturn 2y ago
  If that was true then what do journalists have to be worried about? Why would they need to "fight back" against the LLMs as the above poster said - as you know, ChatGPT can't do original research or investigations on its own. The truth is that the bulk of what is called "journalism" today isn't on-the-ground gumshoes reporting, it's rewording tweets/press releases/PR statements and maybe a couple of phone calls mixed in. A good rule of thumb is that anything that can be replaced by a current-gen AI wasn't worth much to begin with.