3 ms·
OpenAI’s web scraping bot will respect your robots.txt. https://platform.openai.com/docs/gptbot/disallowing-gptbot https://platform.openai.com/docs/gptbot/disa
by joshuanapoli 3y ago
OpenAI’s web scraping bot will respect your robots.txt.
https://platform.openai.com/docs/gptbot/disallowing-gptbot https://platform.openai.com/docs/gptbot/disallowing-gptbot
- nicbou 3y agoIt's a bit late since it already crawled everything once.