5 ms·
Positional preferences, order effects, prompt sensitivity undermine AI judgments
- gizajob 2y ago[flagged]
- giancarlostoro 2y agoI listen to online debates, especially political ones on various platforms, and man. The AI slop that people slap around at each other is beyond horrendous. I would not want an LLM being the final say on something critical. I want the opposite, an LLM should identify things that need follow up review by a qualified person, a person should still confirm the things that "pass" but they can then prioritize what to validate first.
- batshit_beaver 2y agoI don't even trust LLMs enough to spot content that requires validation or nuance.
- giancarlostoro 1y agoLLM's are 100% trust but verify.
- baxtr 2y agoI’d argue real judges are unreliable as well. The real question for me is: are they less reliable than human judges? Probably yes. But I favor a relative measurement to humans than a plain statement like that.
- wetpaws 2y ago[dead]
- nimitkalra 2y agoThere are technical quirks that make LLM judges particularly high variance, sensitive to artifacts in the prompt, and positively/negatively-skewed, as opposed to the subjectivity of human judges. These largely arise from their training distribution and post-training, and can be contained with careful calibration.
- andrewla 2y agoI do think you've hit the heart of the question, but I don't think we can answer the second question. We can measure how unreliable they are, or how susceptible they are to specific changes, just because we can reset them to the same state and run the experiment again. At least for now [1] we do not have that capability with humans, so there's no way to run a matching experiment on humans. The best we can do it is probably to run the limited experiments we can do on humans -- comparing different judge's cross-referenced reliability to get an overall measure and some weak indicator of the reliability of a specific judge based on intra-judge agreement. But when running this on LLMs they would have to keep the previous cases in their context window to get a fair comparison. [1] https://qntm.org/mmacevedo https://qntm.org/mmacevedo
- resource_waste 2y agoI don't study domestic law enough, but I asked a professor of law: "With anything gray, does the stronger/bigger party always win?" He said: "If you ask my students, nearly all of them would say Yes"
- bunderbunder 2y ago> The real question for me is: are they less reliable than human judges? I've spent some time poking at this. I can't go into details, but the short answer is, "Sometimes yes, sometimes no, and it depends A LOT on how you define 'reliable'." My sense is that, the more boring, mechanical and closed-ended the task is, the more likely an LLM is to be more reliable than a human. Because an LLM is an unthinking machine. It doesn't get tired, or hangry, or stressed out about its kid's problems at school. But it's also a doofus with absolutely no common sense whatsoever.
- nimitkalra 2y agoSome other known distributional biases include self-preference bias (gpt-4o prefers gpt-4o generations over claude generations for eg) and structured output/JSON-mode bias [1]. Interestingly, some models have a more positive/negative-skew than others as well. This library [2] also provides some methods for calibrating/stabilizing them. [1]: https://verdict.haizelabs.com/docs/cookbook/distributional-bias https://verdict.haizelabs.com/docs/cookbook/distributional-b... [2]: https://github.com/haizelabs/verdict https://github.com/haizelabs/verdict
- yencabulator 1y agoIt's considered good form on this forum to disclose your affiliation when you advertise for your employer.
- TrackerFF 2y agoI see "panels of judges" mentioned once, but what is the weakness of this? Other than more resource. Worst case you end up with some multi-modal distribution, where two opinions are equal - which seems somewhat unlikely as the panel size grows. Or it could maybe happen in some case with exactly two outcomes (yes/no), but I'd be surprised if such a panel landed on a perfect uniform distribution in its judgments/opinions (50% yes 50% no)
- nimitkalra 2y agoOne method to get a better estimate is to extract the token log-probabilities of "YES" and "NO" from the final logits of the LLM and take a weighted sum [1] [2]. If the LLM is calibrated for your task, there should be roughly a ~50% chance of sampling YES (1) and ~50% chance of NO (0) — yielding 0.5. But generally you wouldn't use a binary outcome when you can have samples that are 50/50 pass/fail. Better to use a discrete scale of 1..3 or 1..5 and specify exactly what makes a sample a 2/5 vs a 4/5, for example You are correct to question the weaknesses of a panel. This class of methods depends on diversity through high-temperature sampling, which can lead to spurious YES/NO responses that don't generalize well and are effectively noise. [1]: https://arxiv.org/abs/2303.16634 https://arxiv.org/abs/2303.16634 [2]: https://verdict.haizelabs.com/docs/concept/extractor/#token-probability https://verdict.haizelabs.com/docs/concept/extractor/#token-...
- shahbaby 2y agoFully agree, I've found that LLMs aren't good at tasks that require evaluation. Think about it, if they were good at evaluation, you could remove all humans in the loop and have recursively self improving AGI. Nice to see an article that makes a more concrete case.
- visarga 2y agoHumans aren't good at validation either. We need tools, experiments, labs. Unproven ideas are a dime a dozen. Remember the hoopla about room temperature superconductivity? The real source of validation is external consequences.
- ken47 2y agoHuman experts set the benchmarks and LLM’s cannot match them in most (maybe any?) fields requiring sophisticated judgment. They are very useful for some things, but sophisticated judgment is not one of them.
- NitpickLawyer 1y agoI think there's more nuance, and the way I read the article is more "beware of these shortcomings", instead of "aren't good". LLM-based evaluation can be good. Several models have by now been trained on previous-gen models used in filtering data and validating RLHF data (pairwise or even more advanced). LLama3 is a good example of this. My take from this article is that there are plenty of gotchas along the way, and you need to be very careful in how you structure your data, and how you test your pipelines, and how you make sure your tests are keeping up with new models. But, like it or not, LLM based evaluation is here to stay. So explorations into this space are good, IMO.
- tempodox 2y ago> We call it 'prompt-engineering' I prefer to call it “prompt guessing”, it's like some modern variant of alchemy.
- BurningFrog 2y ago"Prompt Whispering"?
- th0ma5 2y agoPrompt divining
- thinkling 1y agoImpromptu prompting
- roywiggins 1y agoPrompt dowsing
- layer8 1y agoPrompt vibing
- amlib 1y agoMaybe "prompt mixology" would keep inline with the alchemy theme :)
- wagwang 2y agoCan't wait for the new field of AI psychology
- sidcool 2y agoWe went from Impossible to Unreliable. I like the direction as a techie. But not sure as a sociologist or an anthropologist.
- andrepd 2y ago[flagged]
- dang 2y agoComments like this break the site guidelines, and not just a little. Can you please review https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and take the intended spirit of this site more to heart? Note these: "Please don't fulminate." "Don't be curmudgeonly. Thoughtful criticism is fine, but please don't be rigidly or generically negative." "Please don't sneer, including at the rest of the community." "When disagreeing, please reply to the argument instead of calling names. 'That is idiotic; 1 + 1 is 2, not 3' can be shortened to '1 + 1 is 2, not 3." There's plenty of LLM skepticism on HN and that's fine, but like all comments here, it needs to be thoughtful. (We detached this comment from https://news.ycombinator.com/item?id=44074957 https://news.ycombinator.com/item?id=44074957)
- ken47 2y ago[flagged]
- dang 2y ago"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html (We detached this comment from https://news.ycombinator.com/item?id=44074957 https://news.ycombinator.com/item?id=44074957)
- sanqui 2y agoMeanwhile in Estonia, they just agreed to resolve child support disputes using AI... https://www.err.ee/1609701615/pakosta-enamiku-elatisvaidluste-otsustest-langetab-edaspidi-tehisaru https://www.err.ee/1609701615/pakosta-enamiku-elatisvaidlust...
- suddenlybananas 1y agoMight as well flip a coin.
- alok-g 1y agoNews article: Half our orders would be reversed if there were a higher court: Supreme Court of India https://lawinsider.in/news/half-our-orders-would-be-reversed-if-there-were-a-higher-court-sc https://lawinsider.in/news/half-our-orders-would-be-reversed... If I were to take this literally, it is like a coin flip.
- armchairhacker 2y agoLLMs are good at discovery, since they know a lot, and can retrieve that knowledge from a query that simpler (e.g. regex-based) search engines with the same knowledge couldn't. For example, an LLM that is input a case may discover an obscure law, or notice a pattern in past court cases which establishes precedent. So they can be helpful to a real judge. Of course, the judge must check that the law or precedent aren't hallucinated, and apply to the case in the way the LLM claims. They should also prompt other LLMs and use their own knowledge in case the cited law/precedent contradicts others. There's a similar argument for scientists, mathematicians, doctors, investors, and other fields. LLMs are good at discovery but must be checked.
- amlib 1y agoI would add that "hallucinations" aren't even the only failure mode a LLM can have, it can partially or completely miss what its supposed to find in the discovery process and lead you to believe that there just isn't anything worth pursuing in that particular venue.
- mschuster91 1y ago> it can partially or completely miss what its supposed to find in the discovery process and lead you to believe that there just isn't anything worth pursuing in that particular venue. The problem is that American and UK legal systems never got forced to prune the sometimes centuries-old garbage. And modern Western legal systems tend to have more explicit laws and regulations instead of prior case law, but still, they also accumulate lots of garbage. IMHO, all laws and regulations should come with a set expiry date. If the law or regulation is not renewed, it gets dropped off the book. And for legal systems that have case law, court rulings should expire no later than five years, to force a transition to a system where the law-passing body is forced to work for its money.
- PrimordialEdg71 1y agoLLMs make impressive graders-of-convenience, but their judgments swing wildly with prompt phrasing and option order. Treat them like noisy crowd-raters: randomize inputs, ensemble outputs, and keep a human in the loop whenever single-digit accuracy points matter.
- einrealist 1y ago"and keep a human in the loop whenever single-digit accuracy points matter" So we are supposed to give up on accuracy now? At least with humans (assuming good actors) I can assume an effort for accuracy and correctness. And I can build trust based on some resume and on former interaction. With LLMs, this is more like a coin-flip with each prompt. And since the models are updated constantly, its hard to build some sort of resume. In the meantime, people - especially casual users - might just trust outputs, because its convenient. A single digit error is harder to find. The costs of validating outputs increases with increased accuracy of LLMs. And casual users tend to skip validation because "its a computer and computers are correct". I fear an overall decrease in quality wherever LLMs are included. And any productivity gains are eaten by that.
- nowittyusername 1y agoI've done experiments and basically what I found was that LLM models are extremely sensitive to .....language. Well, duh but let me explain a bit. They will give a different quality/accuracy of answer depending on the system prompt order, language use, length, how detailed the examples are, etc... basically every variable you can think of is responsible for either improving or causing detrimental behavior in the output. And it makes sense once you really grok that LLM;s "reason and think" in tokens. They have no internal world representation. Tokens are the raw layer on which they operate. For example if you ask a bilingual human what their favorite color is, the answer will be that color regardless of what language they used to answer that question. For an LLM, that answer might change depending on the language used, because its all statistical data distribution of tokens in training that conditions the response. Anyway i don't want to make a long post here. The good news out of this is that once you have found the best way in asking questions of your model, you can consistently get accurate responses, the trick is to find the best way to communicate with that particular LLM. That's why i am hard at work on making an auto calibration system that runs through a barrage of ways in finding the best system prompts and other hyperparameters for that specific LLM. The process can be fully automated, just need to set it all up.
- pton_xd 1y agoYep, LLMs tell you "what you want to hear." I can usually predict the response I'll get based on how I phrase the question.
- jonplackett 1y agoI feel like LLMs have a bit of the Clever Hans effect. It takes a lot of my cues as to what it thinks I want it to say or opinion it thinks I want it to have. Clever Hans was a horse who people thought could do maths by tapping his hoof. But actually he was just reading the body language of the person asking the question. Noticing them tense up as he got to the right number of stamps and stopping - still pretty smart for a horse, but the human was still doing the maths!
- 1y ago
- devmor 1y agoIt’s a statistical database of corpuses, not a logic engine. Stop treating LLMs like they are capable of logic, reasoning or judgement. They are not, they never will be. The extent to which they can recall and remix human words to replicate the intent behind those words is an incredible facsimile to thought. It’s nothing short of a mathematical masterpiece. But it’s not intelligence. If it were communicating it’s results in any less human of an interface than conversational, I truly feel that most people would not be so easily fooled into believing it was capable of logic. This doesn’t mean that a facsimile of logic like this has no use. Of course it does, we have seen many uses - some incredible, some dangerous and some pointless - but it is important to always know that there is no thought happening behind the facade. Only a replication of statistically similar communication of thought that may or may not actually apply to your prompt.
- lyu07282 1y agoAlso related: In my observations with tool calling the order of your arguments or fields actually can make a positive or negative effect on performance. You really have to be very careful when constructing your contexts. It doesn't help when all these frameworks and protocols hide these things from you.
- gitroom 1y ago[dead]
- SrslyJosh 1y ago> Positional preferences, order effects, prompt sensitivity undermine AI judgments If you can read between the lines, that says that there's no actual "judgement" going on. If there was a strong logical underpinning to the output, minor differences in input like the phrasing (but not factual content) of a prompt wouldn't make the quality of the output unpredictable.
- Wilsoniumite 1y agoYes and no. People also exhibit these biases, but because degree matters, and because we have no other choice, we still trust them most of the time. That's to say; bias isn't always completely invalidating. I wrote a slightly satirical piece "People are just as bad as my LLMs" here: https://wilsoniumite.com/2025/03/10/people-are-just-as-bad-as-my-llms/ https://wilsoniumite.com/2025/03/10/people-are-just-as-bad-a...
- tiahura 1y agoWord plinko
- ACCount36 1y agoYou could say the same about human "judgement" then. Humans display biases very similar to that of LLMs. This is not a coincidence. LLMs are trained on human-generated data. They attempt to replicate human reasoning - bias and all. There are decisions where "strong logical underpinning" is strong enough to completely drown out the bias and the noise. And then there are decisions that aren't nearly as clear-cut - allowing the bias and the noise to creep into the outcomes. This is true for human and LLM "judgement" both.
- photochemsyn 1y agoDisappointing that they didn't benchmark DeepSeek side by side with OpenAI and Gemini although thelist of funders for cip may explain that: https://www.cip.org/funding-partnerships https://www.cip.org/funding-partnerships Incidentally DeepSeek will give very interesting results if you ask it for a tutorial on prompt engineering - be sure to ask it how to effectively use 'attention anchors' to create 'well-structured prompts', and why rambling disorganized prompts are usually, but not always, detrimental, depending on whether you want 'associative leaps' or not. P.S. I find this intro very useful: > "Task: evaluate the structure of the following prompt in terms of attention anchors and likelihood of it generating a well-structured response. Do not actually reply to the prompt, all we need is an analysis of the structure. Prompt begins:"
- panstromek 1y agoGood, but really none of this should be surprising, given that LLMs are a giant text statistic that generate text based on that statistic. Quirks of that statistic will show up as quirks of the output. When you think about it like that, it doesn't really make sense to assume they have some magical correctness properties. In some sense, they don't classify, they immitate what classification looks like.
- perching_aix 1y ago> In some sense, they don't classify, they immitate what classification looks like. I thought I've seen it all when people decided to consider AI a marketing term and started going off about how current mainstream AI products aren't """"real AI"""", but this is next level.
- panstromek 1y agoI'm not sure I understand your objection (or if it's even an objection), but just to illustrate what I mean - this is literally how the chat interfaces are implemented (or at least initially they were). You're not talking with the model, you're talking with some entity that the model is asked to simulate. The system is just cleverly using your input and the statistic to output something that looks like chat with an assistant. Whether that's real AI or not doesn't really matter. I didn't mean to make it sound like this is not real, just to point out where are the current shortcomings coming from.
- perching_aix 1y agoIt is an objection. I'm not sure if you consider the whole subfield of machine learning that is classification non-existing, or just the fact that LLMs can produce classifications, but either way, I do object. The objection against the former is trivial and self evident, and was more where my sudden upset came from. * Against the latter, the model trying to make the overall text that is its context window approach a chat exchange, by adjusting its own output within it accordingly, doesn't make a hypothetical request for classification in there not performed. You either classify or you don't. If it's doing it in a "misguided" way, it doesn't make it not performed. If it's doing it under the pretense of roleplay, it still doesn't make it not performed. Same is true if the LLM is actually secretly a human operator, or if the LLM is just spewing random tokens. Either you got a classified output of your input or you didn't. It doesn't "look like" anything. I can understand if maybe you mean that the designated notion of the person in the exchange it's trying to approximate for is going to affect the classifications it provides when requested one in the context window, but since these models are trained to "act agentic", I'm not sure if that's a useful thing to ponder (as there's no other way to get anything out of them). I object to the whole "AI is just statistics" notion too. In several situations you want it to do something completely different than what the dataset would support just through rote statistics. That's where you get actual value out of them. One could conveniently recategorize that as just "advanced statistics", or "higher level" statistics, but I think that's a very perverse way of defining statistics. There's very clearly more mathematics involved in LLMs than just statistics. Just the other day, there was a post here trying to regard LLMs as "just topology". Clearly neither of these can be true at the same time, which was consequently explored in the thread too. > You're not talking with the model, you're talking with some entity I'm not suggesting I'm actually talking with anyone or anything in particular beyond the antropomorphization. * What I meant by "people saying AI is not real" is that people claim to regard that the current generation of AI products are not real "artificial intelligences", because they seem to think that it's either "SkyNet" and "Detroit: Become Human", or nothing. Unsurprisingly, these folks don't tend to talk much about OCR, image segmentation and labeling, optical flow, etc. And just like the half a century old field of Artificial Intelligence isn't just some new marketing con that just spawned into existence, classification algorithms aren't some novel snakeoil either. Edit: typing this all out about how I'm aware I'm not actually having conversations with anyone or anything gave me a feeling of realization. This is not good, because intellectually I was always aware of this, meaning I got subconsciously parasocial with these products and services over time. Really concerned about this all of a sudden lol.