3 ms·
OpenAI publishes a list of IP prefixes that its scraper uses, as well as the user-agents: https://platform.openai.com/docs/bots https://platform.openai.com/doc
by peterldowns 2y ago
OpenAI publishes a list of IP prefixes that its scraper uses, as well as the user-agents:
https://platform.openai.com/docs/bots https://platform.openai.com/docs/bots
It does not document how the bot will respond to things like 429 response codes. It does not document how frequently their bots may crawl a given domain/url. It does not document any way to get in touch about abusive crawler practices.
- layer8 2y agoFrom https://pod.geraspora.de/posts/17342163 https://pod.geraspora.de/posts/17342163 (discussed in https://news.ycombinator.com/item?id=42549624 https://news.ycombinator.com/item?id=42549624): “If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really).”