4 ms·
We at pulumi started treating some hallucinations like this as feature requests. Sometimes an llm will hallucination a flag, or option that really makes sense
by adamgordonbell 2y ago
We at pulumi started treating some hallucinations like this as feature requests.
Sometimes an llm will hallucination a flag, or option that really makes sense - it just doesn't actually exist.
- wrs 2y agoThis sort of hallucination happens to me frequently with AWS infrastructure questions. Which is depressing because I can't do anything but agree, "yeah, that API is exactly what any sane person would want, but AWS didn't do that, which is why I'm asking the question".
- miningape 2y agoWhy are you so sure it's what someone sane would want? Maybe there are other ways because there are hidden problems and edge cases with that procedure. It could contradict the fundamental model of the underlying resources but looks correct to someone with a cursory understanding. I'm not saying this is the case, but LLMs are often wrong in subtle ways like this.
- wrs 2y agoWell, I do consider myself sane, and it’s what I want. :) But usually it’s a missing higher-level abstraction, where AWS could have given you the Millenium Falcon kit, but instead it just dumps a bunch of LEGO pieces on your desk and tells you to figure it out.
- kgeist 2y agoAlso, sometimes a flag does exist, but the example places it incorrectly, causing the command to reject it. Or, a flag used to exist but was removed in the latest versions.