3 ms·
OpenAI managed to add this after a lot of complaining, but most AI scrapers lie about their user agent and ignore robots.txt. Plus, OpenAI gets to keep all the
by odo1242 2y ago
OpenAI managed to add this after a lot of complaining, but most AI scrapers lie about their user agent and ignore robots.txt. Plus, OpenAI gets to keep all the data from before they added this.