12 ms·
I have a small physics-based problem I pose to LLMs. It's tricky for humans as well, and all LLMs I've tried (GPT o3, Claude 3.7, Gemini 2.5 Pro) fail to answer
by stavros 2y ago
I have a small physics-based problem I pose to LLMs. It's tricky for humans as well, and all LLMs I've tried (GPT o3, Claude 3.7, Gemini 2.5 Pro) fail to answer correctly. If I ask them to explain their answer, they do get it eventually, but none get it right the first time. Qwen3 with max thinking got it even more wrong than the rest, for what it's worth.
- kenjackson 2y agoYou really had me until the last half of the last sentence.
- stavros 2y agoThe plural of anecdote is data.
- rtaylorgarlock 2y agoOnly in the same way that the plural of 'opinion' is 'fact' ;)
- stavros 2y agoExcept, very literally, data is a collection of single points (ie what we call "anecdotes").
- rwj 2y agoExcept that the plural of anecdotes is definitely not data, because without controlling for confounding variables and sampling biases, you will get garbage.
- scubbo 2y agoGarbage data is still data, and data (garbage or not) is still more valuable than a single anecdote. Insights can only be distilled from data, by first applying those controls you mentioned.
- jimmySixDOF 2y agoOr you can apply the Bezos/Amazon anecdote about anecdotes: At a managers meeting "user stories" about poor support but all the KPIs looked good from the call center so Jeff dials in the number from the meeting speaker phone, gets put on hold, IVR spin cycle, hold again, etc .... His take away was basically "if the data and anecdotes don't match always default to the customer stories".
- fhd2 2y agoBased on my limited understanding of analytics, the data set can be full of biases and anomalies, as long as you find a way to account for them in the analysis, no?
- LegionMammal978 2y agoThe accuracy of your analysis becomes limited to the accuracy of how well you correct for the biases. And it's difficult to measure the bias accurately without lots of good data or cross-examination.
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- bcoates 2y agoNo, Wittgenstein's rule following paradox, Shannon sampling theorem, the law that infinite polynomials pass through any finite set of points (does that have a name?), etc, etc. are all equivalent at the limit to the idea that no amount of anecdotes-per-se add up to anything other than coincidence
- inimino 2y agoNo, no, no. Each of them gives you information.
- whatnow37373 2y agoWithout structural assumptions, there is no necessity - only observed regularity. Necessity literally does not exist. You will never find it anywhere. Hume figured this out quite a while ago and Kant had an interesting response to it. Think the lack of “necessity” is a problem? Try to find “time” or “space” in the data. Data by itself is useless. It’s interesting to see peoples’ reaction to this.
- bijant 2y ago@whatnow37373 — Three sentences and you’ve done what a semester with Kritik der reinen Vernunft couldn’t: made the Hume-vs-Kant standoff obvious. The idea that “necessity” is just the exhaust of our structural assumptions (and that data, naked, can’t even locate time or space) finally snapped into focus. This is exactly the kind of epistemic lens-polishing that keeps me reloading HN.
- cess11 2y agoNo. Anecdote, ἀνέκδοτον, is a story that points to some abstract idea, commonly having something to do with morals. The word means 'not given out'/'not-out-given'. Data is the plural of datum, and arrives in english not from greek, but from latin. The root is however the same as in anecdote, and datum means 'given'. Saying that 'not-given' and 'collection of givens' is the same is clearly nonsensical. A datum has a value and a context in which it was 'given'. What you mean by "points" eludes me, maybe you could elaborate.
- acchow 2y ago“Plural of anecdote is data” is meant to be tongue-in-cheek. Actual data is sampled randomly. Anecdotes very much are not.
- 9rx 2y ago> “Plural of anecdote is data” is meant to be tongue-in-cheek. To be fair, possibly because of this tongue-in-cheek statement, anecdote is sometimes used in place of datum and not as anecdote is usually defined.
- absolutelastone 2y agoone point is a collection of size 1. It is always data.
- 9rx 2y agoTechnically we call it a datum. An anecdote is a story, not a point. But it is true that colloquially anecdote is sometimes used in place of datum.
- WhitneyLand 2y agoThe plural of reliable data is not anecdote.
- deleted 2y ago[deleted]
- tomrod 2y agoDepends on the data generating process.
- WhitneyLand 2y agoOf course, but then you have a system of gathering information with some rigor which is more than merely a collection of anecdotes. That becomes the difference.
- dymk 2y agohttps://en.wikipedia.org/wiki/Thought-terminating_cliche https://en.wikipedia.org/wiki/Thought-terminating_cliche
- wizardforhire 2y agoAhhhahhahahaha stavros is so right but this is such high level bickering I haven't laughed so hard in a long time. Ya’ll are awesome! dymk you deserve a touché for this one. The challenge for sharing data at this stage of the game is that the game is rigged in datas favor. So stavros I hear you. To clarify, if we post our data it’s just going to get fed back into the models making it even harder to vet iterations as they advance.
- tankenmate 2y ago"The plural of anecdote is data.", this is right up there with "1 + 1 = 3, for sufficiently large values of 1". Had an outright genuine guffaw at this one, bravo.
- dataf3l 2y agoI think somebody said it may be 'anecdata'
- windowshopping 2y ago"For what it's worth"? What's wrong with that?
- Jordan-117 2y agoThat's the last third of the sentence.
- phonon 2y agoQwen3-235B-A22B?
- stavros 2y agoYep, on Qwen chat.
- arthurcolle 2y agoHi, I'm starting an evals company, would love to have you as an advisor!
- 999900000999 2y agoNot OP, but what exactly do I need to do. I'll do it for cheap if you'll let me work remote from outside the states.
- refulgentis 2y agoI believe they're kidding, playing on "my singular question isn't answered correctly"
- sieabahlpark 2y ago[dead]
- arthurcolle 2y agoAbsolutely, please reach out! arthur@distributed.systems
- deleted 2y ago[deleted]
- concrete_head 2y agoCan you please share the problem?
- stavros 2y agoI don't really want it added to the training set, but eh. Here you go: > Assume I have a 3D printer that's currently printing, and I pause the print. What expends more energy, keeping the hotend at some temperature above room temperature and heating it up the rest of the way when I want to use it, or turning it completely off and then heat it all the way when I need it? Is there an amount of time beyond which the answer varies? All LLMs I've tried get it wrong because they assume that the hotend cools immediately when stopping the heating, but realize this when asked about it. Qwen didn't realize it, and gave the answer that 30 minutes of heating the hotend is better than turning it off and back on when needed.
- bufferoverflow 2y agoWhat kind of answer do you expect? It all depends on the hotend shape and material, temperature differences, how fast air moves in the room, humidity of the air, etc.
- sampo 2y ago> It all depends on No it doesn't.
- deleted 2y ago[deleted]
- dtech 2y agoKeeping something above room temperature will always use more energy than letting it cool down and heating it back up when needed
- claiir 2y agoSounds like the LLM you used when writing this slop comment struggled with the problem too. :>
- nothink3194 2y ago[dead]
- furyofantares 2y agoI similarly have a small, simple spatial reasoning problem that only reasoning models get right, and not all of them, and which Qwen3 on max reasoning still gets wrong. > I put a coin in a cup and slam it upside-down on a glass table. I can't see the coin because the cup is over it. I slide a mirror under the table and see heads. What will I see if I take the cup (and the mirror) away?
- natrys 2y agoMy first try (omitting chain of thought for brevity): When you remove the cup and the mirror, you will see tails. Here's the breakdown: Setup: The coin is inside an upside-down cup on a glass table. The cup blocks direct view of the coin from above and below (assuming the cup's base is opaque). Mirror Observation: A mirror is slid under the glass table, reflecting the underside of the coin (the side touching the table). If the mirror shows heads, this means the underside of the coin is heads. Coin Orientation: A coin has two sides: heads and tails. If the underside (facing the table) is heads, the top side (facing away from the table) must be tails. Result: Removing the cup reveals the coin on the table. Since the top side was tails all along, you will see tails. Key Insight: The mirror reveals the hidden underside of the coin, but the top side (visible after removing the cup) is its opposite. The mirror’s reflection does not alter the physical orientation of the coin’s sides—it simply reflects the existing state. Answer: Tails.
- SamBam 2y agoHuh, for me it said: Answer: You will see the same side of the coin that you saw in the mirror — heads . Why? The glass table is transparent , so when you look at the coin from below (using a mirror), you're seeing the top side of the coin (the side currently facing up). Mirrors reverse front-to-back , not left-to-right. So the image is flipped in depth, but the orientation of the coin (heads or tails) remains clear. Since the coin hasn't moved during this process, removing the cup and mirror will reveal the exact same face of the coin that was visible via the mirror — which was heads. Final Answer: You will see heads.
- 2y ago
- deleted 2y ago[deleted]
- baxtr 2y agoThis reads like a great story with a tragic ending!
- nopinsight 2y agoCurrent models are quite far away from human-level physical reasoning (paper below). An upcoming version of models trained on world simulation will probably do much better. PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models https://phybench-official.github.io/phybench-demo/ https://phybench-official.github.io/phybench-demo/
- horhay 2y agoThis is more about a physics math aptitude test. You can already see that the best model in math is saturating it halfway. It might not indicate its usefulness in actual physical reasoning, or at the very least, it seems like a bit of a stretch.
- mrkeen 2y agoAs they say, we shouldn't judge AI by the current state-of-the-art, but by how far and fast it's progressing. I can't wait to see future models get it even more wrong than that.
- kaoD 2y agoPersonally (anecdata) I haven't experienced any practical progress in my day-to-day tasks for a long time, no matter how good they became at gaming the benchmarks. They keep being impressive at what they're good at (aggregating sources to solve a very well known problem) and terrible at what they're bad at (actually thinking through novel problems or old problems with few sources). E.g. all ChatGPT, Claude and Gemini were absolutely terrible at generating Liquidsoap[0] scripts. It's not even that complex, but there's very little information to ingest about the problem space, so you can actually tell they are not "thinking". [0] https://www.liquidsoap.info/ https://www.liquidsoap.info/
- prox 2y agoAbsolutely, as soon as they hit that mark where things get really specialized, they start failing a lot. They do generalizations on well documented areas pretty good. I only use it for getting a second opinion as it can search through a lot of documents quickly and find me alternative means.
- Filligree 2y agoThey have broad knowledge, a lot of it, and they work fast. That should be a useful combination- And indeed it is. Essentially every time I buy something these days, I use Deep Research (Gemini 2.5) to first make a shortlist of options. It’s great at that, and often it also points out issues I wouldn’t have thought about. Leave the final decisions to a super slow / smart intelligence (a human), by all means, but for people who claim that LLMs are useless I can only conclude that they haven’t tried very hard.
- jim180 2y agoAbsolutely. All models ar terrible with Objective-C and Swift, compared to let's say JS/HTML/Python. However, I've realized that Claude Code is extremely useful for generating somewhat simple landing pages for some of my projects. It spits out static html+js which is easy to host, with somewhat good looking design. The code isn't the best and to some extent isn't maintainable by a human at all, but it gets the job done.
- animal531 2y agoThey all are using these tests to determine their worth, but to be honest they don't convert well to real world tests. For example I tried Deepseek for code daily over a period of about two months (vs having used ChatGPT before), and its output was terrible. It would produce code with bugs, break existing code when making additions, totally fail at understanding what you're asking etc.
- ggregoryarms 2y agoExactly. If I'm going to be solving bugs, I'd rather they be my own.
- mromanuk 2y agoI was expecting a different outcome, that you tell us that Qwen3 nailed at first.
- laurent_du 2y agoI do the same with a small math problem and so far only Qwen3 got it right (tested all thinking models). So your mileage may vary, as they say!
- claiir 2y agoSame experience with my personal benchmarks. Generally unimpressed with Qwen3.
- throwaway743 2y agoMy favorite test is "Build an MTG Arena Deck in historic format around <strategy_and_or_cards> in <these_colors>. It must be exactly 60 cards and all cards must be from Arena only. Search all sets/cards currently availble on Arena, new and old". Many times they’ll include cards that are only available in paper and/or go over the limit, and when asked to correct a mistake they'll continue to make mistakes. But recently I found that Claude is pretty damn good now at fixing its mistakes and building/optimizing decks for Arena. Asked it to make a deck based on insights it gained from my current decklist, and what it came up with was interesting and pretty fun to play.
- spaceman_2020 2y agoI don’t know about physics, but o3 was able to analyze a floor plan and spot ventilation and circulation issues that even my architect brother wasn’t able to spot in a single glance Maybe it doesn’t make physicists redundant, but it’s definitely making expertise in more mundane domains way more accessible
- the13 2y agoShow prompt