Why would you expect them to be able to given how they work? [1] Someone where I work was trying to get an LLM to evaluate responses to an internal multiple-ch

Why would you expect them to be able to given how they work? [1] Someone where I work was trying to get an LLM to evaluate responses to an internal multiple-choice quiz (A, B or C), putting people into different buckets based on a combination of the total number of correct responses and having answered specific questions correctly. They spent a week "prompt engineering" it back and forth, with subtle changes to their instructions on how the scoring should work, with no appreciable effect on accuracy or consistency. That's another scenario where I felt someone was asking for something with no mechanical sympathy for how it was supposed to happen. Maybe a "thinking" model (why do "AI" companies always abuse terms like this? (rhetorical)) would have been able to get enough stuff into the context for it to be able to get closer to a better outcome, but I took their prompt asked it to write code instead, and got it translated into some overly-commented but simple-enough code which would do the job perfectly every time, including a comment that the instructions they'd provided had a gap where people answering with a certain combination of answers wouldn't fall into any bucket. [1] https://www.youtube.com/watch?v=7xTGNNLPyMI https://www.youtube.com/watch?v=7xTGNNLPyMI