4 ms·

NYTimes, CNN and ABC block OpenAI’s GPTBot web crawler from accessing content

32 points by fuzztester 3y ago

pcurve 3y ago
I wonder what the gptbots dependency on these big media outlets are for training and keeping their dataset fresh.
- TillE 3y ago
  I think even just losing the ability to prompt an LLM with "...in the style of the New York Times" would be interesting. You'd probably end up drawing from parodies of the Times, which is quite different. Of course OpenAI probably still has a full dump of LexisNexis or something.
  - soultrees 3y ago
    But does that include data already scraped? If not, then I don’t see how that will change. What will change however, is a dataset or api for sale, confirmed to be written by human authors which has value for anyone looking to train their model. Future data confirmed not to be written by AI will be invaluable.
  - BoorishBears 3y ago
    You can't lose the style without losing the entire concept of the New York Times. People assume you need ingest a bunch of an author's writing to have the LLM produce writing in that style... I think people seriously underestimate the level of inference these LLMs are capable off. https://chat.openai.com/share/58b38f84-4ddd-4147-989a-c8cbf27a7d34 https://chat.openai.com/share/58b38f84-4ddd-4147-989a-c8cbf2... For example: > Inference: A scholar specializing in narratology might observe that the article employs a classical Aristotelian narrative arc, replete with tension ("nail-biting 45 minutes") and resolution ("heartwarming testament"). This is more typical of storytelling than journalistic reporting, which often employs an "inverted pyramid" structure to relay the most crucial information first. To write an analysis like that, the LLM isn't regurgitating tokens in a way that matches the NYT, it's close to regurgitating academic analyses of writing. You could train the model without a single NYT article and it'd still be able to produce that guidance and then apply it to novel writing pieces to get something NYT-like.
- nvm0n2 3y ago
  Maybe none? These are news sites. ChatGPT still doesn't know about anything that happened after 2021, apparently hundreds of millions of users don't care. The dataset is already very stale but it's still very useful. Also, the lesson of Google News was that news is a commodity. The world is full of websites reporting on the same events and giving those reports away for free. They aren't all going to block GPTBot. And then finally this might even be a good thing for OpenAI. One of the criticisms of ChatGPT is that it has a left wing bias, probably a result of training on left wing biased news media. If NYTimes/CNN/ABC stuff no longer shows up in the training set, ChatGPT's views may come more into line with that of the average person.
mediumsmart 3y ago
It’s a step in the right direction but only the tip of the iceberg when it comes to keeping AI free of pollution.
blitzar 3y ago
I wonder if the GPTBot web crawler respects robot.txt.
- arcanemachiner 3y ago
  Ostensibly, yes. https://platform.openai.com/docs/gptbot/disallowing-gptbot https://platform.openai.com/docs/gptbot/disallowing-gptbot I would not be the slightest bit surprised if some "error" prevented this from working as intended, but hey, what do I know?
  - blitzar 3y ago
    Maybe GPTBot's distant cousin "No user agent" will come visit just a few minutes later.
    - NoZebra120vClip 3y ago
      Yes, I'm skeptical of the headline. Are they blocking a UA or what? It was going to be the LLM-scrapers ignoring robots.txt in order to promote some kind of new standard of discouraging them. The scrapers are all rewriting their privacy policies to just say "all your text is belong to us" so I don't see much recourse for the media outlets, aside from being large with lots of aggressive IP attorneys. And what becomes of all the archive.is pages we're feeding to them anyway?
onetokeoverthe 3y ago
[dead]
aaron695 3y ago
[dead]
ChrisArchitect 3y ago
More related/previous discussion about this when the news started breaking a few weeks ago: Sites scramble to block ChatGPT web crawler after instructions emerge https://news.ycombinator.com/item?id=37094463 https://news.ycombinator.com/item?id=37094463 GPTBot – OpenAI’s Web Crawler https://news.ycombinator.com/item?id=37030568 https://news.ycombinator.com/item?id=37030568

32 points by fuzztester 3y ago

pcurve 3y ago

I wonder what the gptbots dependency on these big media outlets are for training and keeping their dataset fresh.

TillE 3y ago
I think even just losing the ability to prompt an LLM with "...in the style of the New York Times" would be interesting. You'd probably end up drawing from parodies of the Times, which is quite different. Of course OpenAI probably still has a full dump of LexisNexis or something.
- soultrees 3y ago
  But does that include data already scraped? If not, then I don’t see how that will change. What will change however, is a dataset or api for sale, confirmed to be written by human authors which has value for anyone looking to train their model. Future data confirmed not to be written by AI will be invaluable.
- BoorishBears 3y ago
  You can't lose the style without losing the entire concept of the New York Times. People assume you need ingest a bunch of an author's writing to have the LLM produce writing in that style... I think people seriously underestimate the level of inference these LLMs are capable off. https://chat.openai.com/share/58b38f84-4ddd-4147-989a-c8cbf27a7d34 https://chat.openai.com/share/58b38f84-4ddd-4147-989a-c8cbf2... For example: > Inference: A scholar specializing in narratology might observe that the article employs a classical Aristotelian narrative arc, replete with tension ("nail-biting 45 minutes") and resolution ("heartwarming testament"). This is more typical of storytelling than journalistic reporting, which often employs an "inverted pyramid" structure to relay the most crucial information first. To write an analysis like that, the LLM isn't regurgitating tokens in a way that matches the NYT, it's close to regurgitating academic analyses of writing. You could train the model without a single NYT article and it'd still be able to produce that guidance and then apply it to novel writing pieces to get something NYT-like.
nvm0n2 3y ago
Maybe none? These are news sites. ChatGPT still doesn't know about anything that happened after 2021, apparently hundreds of millions of users don't care. The dataset is already very stale but it's still very useful. Also, the lesson of Google News was that news is a commodity. The world is full of websites reporting on the same events and giving those reports away for free. They aren't all going to block GPTBot. And then finally this might even be a good thing for OpenAI. One of the criticisms of ChatGPT is that it has a left wing bias, probably a result of training on left wing biased news media. If NYTimes/CNN/ABC stuff no longer shows up in the training set, ChatGPT's views may come more into line with that of the average person.

mediumsmart 3y ago

It’s a step in the right direction but only the tip of the iceberg when it comes to keeping AI free of pollution.

blitzar 3y ago

I wonder if the GPTBot web crawler respects robot.txt.

arcanemachiner 3y ago
Ostensibly, yes. https://platform.openai.com/docs/gptbot/disallowing-gptbot https://platform.openai.com/docs/gptbot/disallowing-gptbot I would not be the slightest bit surprised if some "error" prevented this from working as intended, but hey, what do I know?
- blitzar 3y ago
  Maybe GPTBot's distant cousin "No user agent" will come visit just a few minutes later.
  - NoZebra120vClip 3y ago
    Yes, I'm skeptical of the headline. Are they blocking a UA or what? It was going to be the LLM-scrapers ignoring robots.txt in order to promote some kind of new standard of discouraging them. The scrapers are all rewriting their privacy policies to just say "all your text is belong to us" so I don't see much recourse for the media outlets, aside from being large with lots of aggressive IP attorneys. And what becomes of all the archive.is pages we're feeding to them anyway?

onetokeoverthe 3y ago

[dead]

aaron695 3y ago

[dead]

ChrisArchitect 3y ago

More related/previous discussion about this when the news started breaking a few weeks ago: Sites scramble to block ChatGPT web crawler after instructions emerge https://news.ycombinator.com/item?id=37094463 https://news.ycombinator.com/item?id=37094463 GPTBot – OpenAI’s Web Crawler https://news.ycombinator.com/item?id=37030568 https://news.ycombinator.com/item?id=37030568