4 ms·
Are these IPs actually from OpenAI/etc. (https://openai.com/gptbot.json https://openai.com/gptbot.json), or is it possibly something else masquerading as these
by Ukv 2y ago
Are these IPs actually from OpenAI/etc. (https://openai.com/gptbot.json https://openai.com/gptbot.json), or is it possibly something else masquerading as these bots? The real GPTBot/Amazonbot/etc. claim to obey robots.txt, and switching to a non-bot UA string seems extra questionable behaviour.
- equestria 2y agoI exclude all the published LLM User-Agents and have a content honeypot on my website. Google obeys, but ChatGPT and Bing still clearly know the content of the honeypot.
- Ukv 2y agoInteresting - do you have a link?
- equestria 2y agoOf course, but I'd rather not share it for obvious reasons. It is a nonsensical biography of a non-existing person.
- jonnycomputer 2y agohow do you determine that they know the content of the honeypot?
- arrowsmith 2y agoPresumably the "honeypot" is an obscured link that humans won't click (e.g. tiny white text on a white background in a forgotten corner of the page) but scrapers will. Then you can determine whether a given IP visited the link.
- 55555 2y agoI interpreted it to mean that a hidden page (linked as u describe) is indexed in Bing or that some "facts" written on a hidden page are regurgitated by ChatGPT.
- jonnycomputer 2y agoI know what a honeypot is, but the question is how the know the scraped data was actually used to train llms. I wondered whether they discovered or verified that by getting the llm to regurgitate content from the honeypot.
- pogue 2y agoWhat's the purpose of the honeypot? Poisoning the LLM or identifying useragents/IPs that shouldn't be seeing it?
- _zagj 2y agoI don't trust OpenAI, and I don't know why anyone else would at this point.