4 ms·
I am observing an extremely high rate of text hallucinations with gpt-4o (gpt-4o-2024-05-13) as tested via the API. I advise extreme caution with it. In contras
by OutOfHere 3y ago
I am observing an extremely high rate of text hallucinations with gpt-4o (gpt-4o-2024-05-13) as tested via the API. I advise extreme caution with it. In contrast, I see no such concern with gpt-4-turbo-preview (gpt-4-0125-preview).
- fdb 3y agoSame here. I observed it making up functions in d3 (`d3.geoProjectionRaw` and `d3.geoVisible`), in addition to ignoring functions it could have used.
- bigyikes 3y agoIf true, makes me wonder what kind of regression testing OpenAI does for these models. It can’t be easy to write a unit test for hallucinations.
- OutOfHere 3y agoAt a high level, ask it to produce a ToC of information about something that you know will exist in the future, but does not yet exist, but also tell it to decline the request if it doesn't verifiably know the answer.
- bigyikes 3y agoHow do you generalize that for all inputs though?
- OutOfHere 3y agoI am not sure I understand the question. I sampled various topics. I used this prompt: https://raw.githubusercontent.com/impredicative/podgenai/master/prompts/list_subtopics.txt https://raw.githubusercontent.com/impredicative/podgenai/mas... In the prompt, substitute {topic} with something from the near future. As I noted, it behaves correctly for turbo (rejecting the request), and very badly for o (hallucinating nonsense).
- Seattle3503 3y agoI asked it to describe the books of an obscure author. GPT-4o hallucinated books. GPT-4 knew it needed to do an internet search.