3 ms·
As the answer would seem to be 'no' (crawlers can ignore robots.txt rules), I'm wondering if GPT is going to usher in an era where new content is simply not mad
by MontgomeryPi2 4y ago
As the answer would seem to be 'no' (crawlers can ignore robots.txt rules), I'm wondering if GPT is going to usher in an era where new content is simply not made available to view/crawl on the web. E.g. Want to know all the cool events happening this weekend in NYC? Ask our vertical gpt-chat site to find out.
- speedgoose 4y agoOpenAI uses the common crawl dataset and I would expect them to respect the robots.txt rules.
- JohnFen 4y ago> I'm wondering if GPT is going to usher in an era where new content is simply not made available to view/crawl on the web. I've stopped adding new stuff to the public areas of my websites. Not as a result of GPT specifically, but as a result of a dramatic increase in the amount of scraping being done to support things that I don't approve of. AI training in general being one of them.