I just tested ChatGPT 4o and Claude 3.5 Sonnet using the following prompt, with and without the last sentence: “Write a one-paragraph summary of research since

I just tested ChatGPT 4o and Claude 3.5 Sonnet using the following prompt, with and without the last sentence: “Write a one-paragraph summary of research since 2015 on the use of machine translation in second-language academic writing. Include five in-text citations to previous research, and include full reference information for each citation after the paragraph. Do not hallucinate any of the citations and references.” When the last sentence was not included, four out of five of ChatGPT’s references were hallucinated; the fifth referred to an existing paper but had an incorrect year. All of Claude’s references were hallucinated. When I included the last sentence, ChatGPT did a web search and came up with five genuine citations. Claude, which cannot do web searches, apologized that it could not write the paragraph or provide the references. Maybe a better test of the effectiveness of "Do not hallucinate" would be prompts that might lead to hallucinated claims that the LLM cannot check using web searches, the execution of Python programs, or other external processes. Any ideas?