7 ms·

I like to make up my own tests, that way you know it is actually thinking. Tests that require thinking about the physical world are the most revealing. My new

I like to make up my own tests, that way you know it is actually thinking. Tests that require thinking about the physical world are the most revealing. My new favourite is: You have 2 minutes to cool down a cup of coffee to the lowest temp you can. You have two options: 1. Add cold milk immediately, then let it sit for 2 mins. 2. Let it sit for 2 mins, then add cold milk. Which one cools the coffee to the lowest temperature and why? Phrased this way without any help, all but the thinking models get it wrong

akoboldfrying 2y ago
The fact that the answer is interesting makes me suspect that it's not a good test for thinking. I remember reading the explanation for the answer somewhere on the internet years ago, and it's stayed with me ever since. It's interesting enough that it's probably been written about multiple times in multiple places. So I think it would probably stay with a transformer trained on large volumes of data from the internet too. I think a better test of thinking is to provide detail about something so mundane and esoteric that no one would have ever thought to communicate it to other people for entertainment, and then ask it a question about that pile of boring details.
- xx_ns 2y ago
  Out of curiosity, what is the answer? From your comment, it seems like the more obvious choice is the incorrect one. EDIT: By the more obvious one, I mean letting it cool and then adding milk. As the temperature difference between the coffee and the surrounding air is higher, the coffee cools down faster. Is this wrong?
  - danbruc 2y ago
    That is the correct answer. Also there is a lot of potential nuance, like evaporation or when you take the milk out of the fridge or the specific temperatures of everything, but under realistic settings adding the milk late will get you the colder coffee.
    - ac2u 2y ago
      Does the ceramic mug become a factor? As in adding milk first allows the milk to absorb heat that otherwise would have been stored in the mug too quickly and then radiate back into the liquid over time slowing its cooling curve. (I have no idea btw I just enjoy trying to come up with gotchas)
  - xeyownt 2y ago
    I'd say adding milk late is the best. You have coffee with volume and heat V and Q, milk v and q. Whatever you do, you'll get volume v+V and heat Q+q. Q can become Q' if you let it cool down first, or (Q+q)' if you add the milk first then let it cool down. But because milk is cold, the Q/V > (Q+q)/(V+v), hence the loss Q -> Q' is bigger than (Q+q) -> (Q+q)'. The best answer though is to put the coffee on a plate, and forget about the milk.
  - chairdoor 2y ago
    Isn't the answer milk first, then let sit? You only have 2 minutes, so if you're adding the milk after 2 minutes have already elapsed, then you've already exceeded the time limit, meaning the final measurement would take place before the milk is even poured in.
    - jonplackett 2y ago
      Adding the milk second is colder. The bigger the temp difference the more cooling. So by putting the milk in right away you make the temp difference between the surroundings and the coffee smaller = less cooling over your 2 mins. I like puzzles with a satisfying answer
      - genewitch 2y ago
        Parent is complaining about being technically hotter because the time the temperature is read for logging is at 120seconds. I notice this on HN more than places like reddit and Mastodon. I think it's a benefit when writing requirements and programming to specs. It's less of a benefit when having a freeform discussion.
- s-macke 2y ago
  Simple Bench goes in this direction: https://simple-bench.com/ https://simple-bench.com/
  - sheepdestroyer 2y ago
    Yet Another Benchmark, great I love benchmarks(!) but, will this page be kept up2date?
    - s-macke 2y ago
      Yes, permanently. Sonnet 3.7 is already number one in the ranking. Grok3 has no API yet.
danbruc 2y ago
No need for thinking, that question can be found discussed and explained many times online and has almost certainly been part of the training data.
- jonplackett 2y ago
  The fact that all the models I’ve tried except the thinking ones get it wrong suggests not. They get caught up in the idea that adding milk first cools it fastest and can’t escape from that
  - cristiancavalli 2y ago
    First page of Google search results from 7 years ago: https://www.quora.com/You-have-2-cups-of-coffee-50-degrees-warm-You-immediately-put-10-deg-cold-milk-into-one-and-after-10-minutes-the-same-amount-of-milk-into-the-other-Which-coffee-will-be-colder-after-10-minutes https://www.quora.com/You-have-2-cups-of-coffee-50-degrees-w... People making up their own benchmarks for these things has confirmed one thing for me: The bias that people think they mostly have original thoughts is extremely strong. I find if I have a “good” idea someone has probably already thought of it as well and maybe even written about it. About 0.01% of the time do I have an idea that one may consider novel and even that’s probably my own bias and overstated. This example just confirms that these models don’t really seem to reason and have a really hard time doing the basic generalization they can with fewer examples.
gwd 2y ago
> Phrased this way without any help, all but the thinking models get it wrong I C&P'd it into Claude 3.7 with thinking, and it gave the correct answer (which I'm pretty sure is #2). Including the CoT, where it actually does math (which I haven't checked), and final response. # THINKING Let's analyze the two options. Option 1: Add cold milk immediately, then let it sit for 2 mins. Option 2: Let it sit for 2 mins, then add cold milk. To determine which option leads to the lowest final temperature, I need to consider the physics of cooling. First, let's understand how cooling works in these scenarios: 1. When a hot liquid sits, it cools according to Newton's Law of Cooling, which states that the rate of heat loss is proportional to the temperature difference between the object and its surroundings. - The rate of cooling is faster when the temperature difference is greater - As the liquid cools, the rate of cooling decreases 2. When we add cold milk to hot coffee, we get an immediate temperature drop due to mixing liquids of different temperatures. The final temperature after mixing can be calculated using the principle of thermal equilibrium. Now, let's analyze each option: Option 1: Add cold milk immediately, then let it sit for 2 mins - The coffee temperature drops immediately when milk is added - Then this coffee+milk mixture cools for 2 minutes - The cooling rate will be based on the temperature difference between the coffee+milk mixture and room temperature Option 2: Let it sit for 2 mins, then add cold milk - The black coffee cools for 2 minutes - Then milk is added, causing an immediate drop in temperature - The cooling rate during the 2 minutes will be based on the temperature difference between the black coffee and room temperature The key insight is that hot coffee cools faster than warm coffee because the rate of cooling is proportional to the temperature difference. Since black coffee starts at a higher temperature than coffee with milk, it will lose more heat energy during the 2-minute cooling period. Let's make this more concrete with some theoretical numbers: - Let's say black coffee starts at 90°C - Room temperature is 20°C - Cold milk is at 5°C and makes up 10% of the final volume Option 1: - Adding milk immediately might bring the temperature to about 81.5°C (90 × 0.9 + 5 × 0.1) - This mixture then cools for 2 minutes from 81.5°C to some lower temperature, let's call it T1 Option 2: - The black coffee cools for 2 minutes from 90°C to some temperature, let's call it T2 - Then we add milk, which further reduces the temperature to 0.9 × T2 + 0.1 × 5 The question is: which is lower, T1 or 0.9 × T2 + 0.1 × 5? According to Newton's Law of Cooling, the black coffee in Option 2 will lose more heat during the 2 minutes because it starts at a higher temperature and thus has a greater temperature difference with the surroundings. This means the drop from 90°C to T2 will be greater than the drop from 81.5°C to T1. Let's think about this mathematically: According to Newton's Law of Cooling: T(t) = Ts + (T0 - Ts) × e^(-kt) Where: - T(t) is the temperature at time t - Ts is the surrounding temperature - T0 is the initial temperature - k is the cooling constant For Option 1: T1 = 20 + (81.5 - 20) × e^(-k × 2) = 20 + 61.5 × e^(-2k) For Option 2, before adding milk: T2 = 20 + (90 - 20) × e^(-k × 2) = 20 + 70 × e^(-2k) After adding milk in Option 2, the temperature would be: 0.9 × T2 + 0.1 × 5 = 0.9 × (20 + 70 × e^(-2k)) + 0.5 = 18 + 63 × e^(-2k) + 0.5 = 18.5 + 63 × e^(-2k) Now we compare: T1 = 20 + 61.5 × e^(-2k) Final temp in Option 2 = 18.5 + 63 × e^(-2k) For any positive value of k (which it must be for cooling), e^(-2k) is between 0 and 1. If e^(-2k) = 1 (no cooling at all): T1 = 20 + 61.5 = 81.5 Final temp in Option 2 = 18.5 + 63 = 81.5 (They would be the same) If e^(-2k) = 0 (instant cooling to room temperature): T1 = 20 Final temp in Option 2 = 18.5 (Option 2 would be better) For values in between, we need to compare 20 + 61.5 × e^(-2k) vs 18.5 + 63 × e^(-2k) Subtracting: (T1 - Final temp in Option 2) = 1.5 - 1.5 × e^(-2k) This will be positive (meaning T1 is higher) when e^(-2k) < 1, which is true for all positive k. So, mathematically, Option 2 (letting it sit, then adding milk) results in a lower final temperature. This makes physical sense because: 1. The hotter black coffee loses more heat during the 2 minutes than the cooler coffee+milk mixture would 2. The cold milk provides the same cooling effect whether added at the beginning or end Therefore, Option 2 (letting the coffee sit for 2 minutes, then adding cold milk) will result in the lowest final temperature. # ANSWER # Coffee Cooling Analysis ## Option 2 wins: Let it sit for 2 mins, then add cold milk This will give you the lowest final temperature, and here's why: ### The physics behind it: 1. *Newton's Law of Cooling*: The rate at which something cools is proportional to the temperature difference between the object and its surroundings. 2. *Key insight*: Hotter objects cool faster than cooler ones because the temperature gap with the environment is larger. ### Comparing the options: *Option 1 (Add milk first, then wait):* - Milk immediately lowers coffee temperature - The coffee+milk mixture cools for 2 minutes, but at a slower rate because it's already cooler *Option 2 (Wait 2 mins, then add milk):* - Black coffee starts hotter and cools faster during the 2 minutes - More total heat is lost during the waiting period - Then milk provides the same temperature drop at the end The black coffee's higher initial temperature difference with the room causes it to lose more total heat energy during the waiting period, making Option 2 the more effective cooling strategy.
freehorse 2y ago
I asked this to QwQ and it started writing equations (newton's law) and arrived at T_2 < T_1, then said this is counterintuitive, started writing more equations and arrived to the same, starts writing an explanation on why this is indeed the case instead of what it is intuitive, and concludes to the right answer. It is the only model I gave this and actually approached it by writing math. Usually I am not that impressed with reasoning models, but this was quite fun to watch.
ur-whale 2y ago
> I like to make up my own tests You just ruined your own test by publishing it on the internets
- matt-attack 2y ago
  Yeah, but he didn’t post the answer.
iamnotagenius 2y ago
[dead]
pythonaut_16 2y ago
I’m not sure how much this tells me about a model’s coding ability though. It might correlate to design level thinking but it also might not.
mrcwinn 2y ago
Obviously you would prepare cold brew the night before.
vintermann 2y ago
I have another easy one which thinking models get wrong: "Anhentafel numbers start with you as 1. To find the Ahhentafel number of someone's father, double it. To find the Ahnentafel number of someone's mother, double it and add one. Men pass on X chromosome DNA to their daughters, but none to their sons. Women pass on X chromosome DNA to both their sons and daughters. List the Ahnentafel numbers of the closest 20 ancestors a man may have inherited X DNA from." For smaller models, it's probably fair to change the question to something like: "Could you have inherited X chromosome DNA from your ancestor with Ahnentafel number 33? Does the answer to that question depend on whether you are a man or a woman?" They still fail.
- audiodude 2y ago
  Yeah I wouldn't call this easy...
  - vintermann 2y ago
    You can just do it generation for generation. The only thing hard about it is that it's two explained concepts you need to combine. A model which aces math Olympiad problems shouldn't have any trouble with this whatsoever - unless it's overfitting on them somehow.
atlex2 2y ago
Yes absolutely this! We're working on these problems at FlyShirley for our pilot training tool. My go-to is: I'm facing 160 degrees and want to face north. What's the quickest way to turn and by how much? For small models and when attention is "taken up", these sorts of questions really send a model for a loop. Agreed - especially noticeable with small reasoning models.
- KiwiJohnno 2y ago
  I just tried this with a smaller "thinking" model (deepseek distill, running locally) and boy are you right. It keeps flipping between which direction it should turn, second guessing its thought process and then getting sidetracked with a different approach.
jonplackett 2y ago
Can’t edit my comment - this was meant to say NON-thinking models can’t do it. The thinking ones can and that’s what is interesting (face palm)
- tanaykarnik 2y ago
  > all *but* the thinking models get it wrong don't worry it's "all but", so you are fine.
astrodude 2y ago
how do you know that this test doesn't already exist somewhere on the internet and LLMs were trained on it?

akoboldfrying 2y ago

The fact that the answer is interesting makes me suspect that it's not a good test for thinking. I remember reading the explanation for the answer somewhere on the internet years ago, and it's stayed with me ever since. It's interesting enough that it's probably been written about multiple times in multiple places. So I think it would probably stay with a transformer trained on large volumes of data from the internet too. I think a better test of thinking is to provide detail about something so mundane and esoteric that no one would have ever thought to communicate it to other people for entertainment, and then ask it a question about that pile of boring details.

xx_ns 2y ago
Out of curiosity, what is the answer? From your comment, it seems like the more obvious choice is the incorrect one. EDIT: By the more obvious one, I mean letting it cool and then adding milk. As the temperature difference between the coffee and the surrounding air is higher, the coffee cools down faster. Is this wrong?
- danbruc 2y ago
  That is the correct answer. Also there is a lot of potential nuance, like evaporation or when you take the milk out of the fridge or the specific temperatures of everything, but under realistic settings adding the milk late will get you the colder coffee.
  - ac2u 2y ago
    Does the ceramic mug become a factor? As in adding milk first allows the milk to absorb heat that otherwise would have been stored in the mug too quickly and then radiate back into the liquid over time slowing its cooling curve. (I have no idea btw I just enjoy trying to come up with gotchas)
- xeyownt 2y ago
  I'd say adding milk late is the best. You have coffee with volume and heat V and Q, milk v and q. Whatever you do, you'll get volume v+V and heat Q+q. Q can become Q' if you let it cool down first, or (Q+q)' if you add the milk first then let it cool down. But because milk is cold, the Q/V > (Q+q)/(V+v), hence the loss Q -> Q' is bigger than (Q+q) -> (Q+q)'. The best answer though is to put the coffee on a plate, and forget about the milk.
- chairdoor 2y ago
  Isn't the answer milk first, then let sit? You only have 2 minutes, so if you're adding the milk after 2 minutes have already elapsed, then you've already exceeded the time limit, meaning the final measurement would take place before the milk is even poured in.
  - jonplackett 2y ago
    Adding the milk second is colder. The bigger the temp difference the more cooling. So by putting the milk in right away you make the temp difference between the surroundings and the coffee smaller = less cooling over your 2 mins. I like puzzles with a satisfying answer
    - genewitch 2y ago
      Parent is complaining about being technically hotter because the time the temperature is read for logging is at 120seconds. I notice this on HN more than places like reddit and Mastodon. I think it's a benefit when writing requirements and programming to specs. It's less of a benefit when having a freeform discussion.
s-macke 2y ago
Simple Bench goes in this direction: https://simple-bench.com/ https://simple-bench.com/
- sheepdestroyer 2y ago
  Yet Another Benchmark, great I love benchmarks(!) but, will this page be kept up2date?
  - s-macke 2y ago
    Yes, permanently. Sonnet 3.7 is already number one in the ranking. Grok3 has no API yet.

danbruc 2y ago

No need for thinking, that question can be found discussed and explained many times online and has almost certainly been part of the training data.

jonplackett 2y ago
The fact that all the models I’ve tried except the thinking ones get it wrong suggests not. They get caught up in the idea that adding milk first cools it fastest and can’t escape from that
- cristiancavalli 2y ago
  First page of Google search results from 7 years ago: https://www.quora.com/You-have-2-cups-of-coffee-50-degrees-warm-You-immediately-put-10-deg-cold-milk-into-one-and-after-10-minutes-the-same-amount-of-milk-into-the-other-Which-coffee-will-be-colder-after-10-minutes https://www.quora.com/You-have-2-cups-of-coffee-50-degrees-w... People making up their own benchmarks for these things has confirmed one thing for me: The bias that people think they mostly have original thoughts is extremely strong. I find if I have a “good” idea someone has probably already thought of it as well and maybe even written about it. About 0.01% of the time do I have an idea that one may consider novel and even that’s probably my own bias and overstated. This example just confirms that these models don’t really seem to reason and have a really hard time doing the basic generalization they can with fewer examples.

gwd 2y ago

> Phrased this way without any help, all but the thinking models get it wrong I C&P'd it into Claude 3.7 with thinking, and it gave the correct answer (which I'm pretty sure is #2). Including the CoT, where it actually does math (which I haven't checked), and final response. # THINKING Let's analyze the two options. Option 1: Add cold milk immediately, then let it sit for 2 mins. Option 2: Let it sit for 2 mins, then add cold milk. To determine which option leads to the lowest final temperature, I need to consider the physics of cooling. First, let's understand how cooling works in these scenarios: 1. When a hot liquid sits, it cools according to Newton's Law of Cooling, which states that the rate of heat loss is proportional to the temperature difference between the object and its surroundings. - The rate of cooling is faster when the temperature difference is greater - As the liquid cools, the rate of cooling decreases 2. When we add cold milk to hot coffee, we get an immediate temperature drop due to mixing liquids of different temperatures. The final temperature after mixing can be calculated using the principle of thermal equilibrium. Now, let's analyze each option: Option 1: Add cold milk immediately, then let it sit for 2 mins - The coffee temperature drops immediately when milk is added - Then this coffee+milk mixture cools for 2 minutes - The cooling rate will be based on the temperature difference between the coffee+milk mixture and room temperature Option 2: Let it sit for 2 mins, then add cold milk - The black coffee cools for 2 minutes - Then milk is added, causing an immediate drop in temperature - The cooling rate during the 2 minutes will be based on the temperature difference between the black coffee and room temperature The key insight is that hot coffee cools faster than warm coffee because the rate of cooling is proportional to the temperature difference. Since black coffee starts at a higher temperature than coffee with milk, it will lose more heat energy during the 2-minute cooling period. Let's make this more concrete with some theoretical numbers: - Let's say black coffee starts at 90°C - Room temperature is 20°C - Cold milk is at 5°C and makes up 10% of the final volume Option 1: - Adding milk immediately might bring the temperature to about 81.5°C (90 × 0.9 + 5 × 0.1) - This mixture then cools for 2 minutes from 81.5°C to some lower temperature, let's call it T1 Option 2: - The black coffee cools for 2 minutes from 90°C to some temperature, let's call it T2 - Then we add milk, which further reduces the temperature to 0.9 × T2 + 0.1 × 5 The question is: which is lower, T1 or 0.9 × T2 + 0.1 × 5? According to Newton's Law of Cooling, the black coffee in Option 2 will lose more heat during the 2 minutes because it starts at a higher temperature and thus has a greater temperature difference with the surroundings. This means the drop from 90°C to T2 will be greater than the drop from 81.5°C to T1. Let's think about this mathematically: According to Newton's Law of Cooling: T(t) = Ts + (T0 - Ts) × e^(-kt) Where: - T(t) is the temperature at time t - Ts is the surrounding temperature - T0 is the initial temperature - k is the cooling constant For Option 1: T1 = 20 + (81.5 - 20) × e^(-k × 2) = 20 + 61.5 × e^(-2k) For Option 2, before adding milk: T2 = 20 + (90 - 20) × e^(-k × 2) = 20 + 70 × e^(-2k) After adding milk in Option 2, the temperature would be: 0.9 × T2 + 0.1 × 5 = 0.9 × (20 + 70 × e^(-2k)) + 0.5 = 18 + 63 × e^(-2k) + 0.5 = 18.5 + 63 × e^(-2k) Now we compare: T1 = 20 + 61.5 × e^(-2k) Final temp in Option 2 = 18.5 + 63 × e^(-2k) For any positive value of k (which it must be for cooling), e^(-2k) is between 0 and 1. If e^(-2k) = 1 (no cooling at all): T1 = 20 + 61.5 = 81.5 Final temp in Option 2 = 18.5 + 63 = 81.5 (They would be the same) If e^(-2k) = 0 (instant cooling to room temperature): T1 = 20 Final temp in Option 2 = 18.5 (Option 2 would be better) For values in between, we need to compare 20 + 61.5 × e^(-2k) vs 18.5 + 63 × e^(-2k) Subtracting: (T1 - Final temp in Option 2) = 1.5 - 1.5 × e^(-2k) This will be positive (meaning T1 is higher) when e^(-2k) < 1, which is true for all positive k. So, mathematically, Option 2 (letting it sit, then adding milk) results in a lower final temperature. This makes physical sense because: 1. The hotter black coffee loses more heat during the 2 minutes than the cooler coffee+milk mixture would 2. The cold milk provides the same cooling effect whether added at the beginning or end Therefore, Option 2 (letting the coffee sit for 2 minutes, then adding cold milk) will result in the lowest final temperature. # ANSWER # Coffee Cooling Analysis ## Option 2 wins: Let it sit for 2 mins, then add cold milk This will give you the lowest final temperature, and here's why: ### The physics behind it: 1. *Newton's Law of Cooling*: The rate at which something cools is proportional to the temperature difference between the object and its surroundings. 2. *Key insight*: Hotter objects cool faster than cooler ones because the temperature gap with the environment is larger. ### Comparing the options: *Option 1 (Add milk first, then wait):* - Milk immediately lowers coffee temperature - The coffee+milk mixture cools for 2 minutes, but at a slower rate because it's already cooler *Option 2 (Wait 2 mins, then add milk):* - Black coffee starts hotter and cools faster during the 2 minutes - More total heat is lost during the waiting period - Then milk provides the same temperature drop at the end The black coffee's higher initial temperature difference with the room causes it to lose more total heat energy during the waiting period, making Option 2 the more effective cooling strategy.

freehorse 2y ago

I asked this to QwQ and it started writing equations (newton's law) and arrived at T_2 < T_1, then said this is counterintuitive, started writing more equations and arrived to the same, starts writing an explanation on why this is indeed the case instead of what it is intuitive, and concludes to the right answer. It is the only model I gave this and actually approached it by writing math. Usually I am not that impressed with reasoning models, but this was quite fun to watch.

ur-whale 2y ago

> I like to make up my own tests You just ruined your own test by publishing it on the internets

matt-attack 2y ago
Yeah, but he didn’t post the answer.

iamnotagenius 2y ago

[dead]

pythonaut_16 2y ago

I’m not sure how much this tells me about a model’s coding ability though. It might correlate to design level thinking but it also might not.

mrcwinn 2y ago

Obviously you would prepare cold brew the night before.

vintermann 2y ago

I have another easy one which thinking models get wrong: "Anhentafel numbers start with you as 1. To find the Ahhentafel number of someone's father, double it. To find the Ahnentafel number of someone's mother, double it and add one. Men pass on X chromosome DNA to their daughters, but none to their sons. Women pass on X chromosome DNA to both their sons and daughters. List the Ahnentafel numbers of the closest 20 ancestors a man may have inherited X DNA from." For smaller models, it's probably fair to change the question to something like: "Could you have inherited X chromosome DNA from your ancestor with Ahnentafel number 33? Does the answer to that question depend on whether you are a man or a woman?" They still fail.

audiodude 2y ago
Yeah I wouldn't call this easy...
- vintermann 2y ago
  You can just do it generation for generation. The only thing hard about it is that it's two explained concepts you need to combine. A model which aces math Olympiad problems shouldn't have any trouble with this whatsoever - unless it's overfitting on them somehow.

atlex2 2y ago

Yes absolutely this! We're working on these problems at FlyShirley for our pilot training tool. My go-to is: I'm facing 160 degrees and want to face north. What's the quickest way to turn and by how much? For small models and when attention is "taken up", these sorts of questions really send a model for a loop. Agreed - especially noticeable with small reasoning models.

KiwiJohnno 2y ago
I just tried this with a smaller "thinking" model (deepseek distill, running locally) and boy are you right. It keeps flipping between which direction it should turn, second guessing its thought process and then getting sidetracked with a different approach.

jonplackett 2y ago

Can’t edit my comment - this was meant to say NON-thinking models can’t do it. The thinking ones can and that’s what is interesting (face palm)

tanaykarnik 2y ago
> all *but* the thinking models get it wrong don't worry it's "all but", so you are fine.

astrodude 2y ago

how do you know that this test doesn't already exist somewhere on the internet and LLMs were trained on it?