4 ms·

> OpenAI posted today that ChatGPT can once more trawl the web for current information, offering answers taken directly from “current and authoritative” sources

by koolba 3y ago

> OpenAI posted today that ChatGPT can once more trawl the web for current information, offering answers taken directly from “current and authoritative” sources, which it cites in its responses. The list is apparently cited in responses, but is the list of possible sources itself public?

simonw 3y ago
I imagine it's everything on the web that's been crawled by Bing and hasn't blocked ChatGPT explicitly in its robots.txt file.
- willsmith72 3y ago
  have openai's crawlers really paid attention to people's robots.txts?
  - simonw 3y ago
    Yes. There's even a project tracking which newspapers are blocking them here: https://palewi.re/docs/news-homepages/openai-gptbot-robotstxt.html https://palewi.re/docs/news-homepages/openai-gptbot-robotstx...
    - input_sh 3y ago
      Oh wow, getting 43.6% of newspapers to do anything collectively over a brief period of time is... definitely not an easy achievement.
      - solardev 3y ago
        Heh, they probably don't want a repeat of Google News
      - is_true 3y ago
        USA today has a lot of newspapers for example. Not sure what % of that list belongs to them nor what's the source.
  - theshackleford 3y ago
    It would appear so from the significant network of sites I am responsible for that have put such robots.txt changes into place.
singularity2001 3y ago
web trawling is the shallow version of web crawling? that's genius I never heard that before