5 ms·

Ok. My go to puzzle is this: You have 2 minutes to cool down a cup of coffee to the lowest temp you can You have two options: 1. Add cold milk immediately, t

Ok. My go to puzzle is this: You have 2 minutes to cool down a cup of coffee to the lowest temp you can You have two options: 1. Add cold milk immediately, then let it sit for 2 mins. 2. Let it sit for 2 mins, then add the cold milk. Which one cools the coffee to the lowest temperature and why? And Mercury gets this right - while as of right now ChatGPT 4o get it wrong. So that’s pretty impressive.

byearthithatius 1y ago
Token-based Gemini is 4 seconds of thinking: Okay, let's break this down using the principle of heat transfer (specifically Newton's Law of Cooling): Heat loss is faster when the temperature difference is greater. A very hot object loses heat to cool surroundings much faster than a warm object. Option 1: Add milk immediately. You instantly lower the coffee's temperature by mixing it with cold milk. Now, this warmer (not hot) mixture sits for 2 minutes. Because the temperature difference between the mixture and the room is smaller, it cools down more slowly over those 2 minutes. Option 2: Let it sit for 2 mins, then add milk. The very hot coffee sits for 2 minutes. Because the temperature difference between the hot coffee and the room is large, it loses heat more quickly during these 2 minutes. After 2 minutes of rapid cooling, you then add the cold milk, lowering the temperature further. Conclusion: To get the coffee to the lowest temperature, you should choose Option 2: Let it sit for 2 mins, then add the cold milk.
crazygringo 1y ago
For me, ChatGPT (the free version, GPT-4o mini I believe?) gets it right, choosing option 2 because the coffee will cool faster due to the larger temperature difference. Unless there's a gotcha somewhere in your prompt that I'm missing, like what if the temperature of the room is hotter than the coffee, or so cold that the coffee becomes colder than the milk, or something? I would be suprised if any models get it wrong, since I assume it shows up in training data a bunch?
- deleted 1y ago
  [deleted]
- jonplackett 1y ago
  This is what I got from full-fat 4o. Maybe thinking less helps! ChatGPT: Option 1 — Add the cold milk immediately — will result in a lower final temperature after 2 minutes. Why: • Heat loss depends on the temperature difference between the coffee and the environment (usually room temperature). • If you add the milk early, the overall temperature of the coffee-milk mixture is reduced immediately. This lowers the average temperature over the 2 minutes, so less heat is lost to the air. • If you wait 2 minutes to add the milk, the hotter coffee loses more heat to the environment during those 2 minutes, but when you finally add the milk, it doesn’t cool it as much because the coffee’s already cooler and the temp difference between the milk and the coffee is smaller. Summary: • Adding milk early = cooler overall drink after 2 minutes. • Adding milk late = higher overall temp after 2 minutes, because more heat escapes during the time the coffee is hotter. Want me to show a simple simulation or visualisation of this?
  - crazygringo 1y ago
    Oof. I wonder what makes it so bad? In my experience LLM's tend to be pretty good at basic logic as long as they understand the domain well enough. I mean, it even gets it right at first -- "This lowers the average temperature over the 2 minutes, so less heat is lost to the air." -- but then it seems to get conceptually confused about heat loss vs cooling, which is surprising.
maytc 1y ago
That example is probably in the training data? The puzzle assumes that the room temperature is greater than the cold milk's temperature. When I added that the room temperature is, say, -10 °C, Mercury fails to see the difference.
- djrj477dhsnv 1y ago
  Under any reasonable assumptions for the size and shape of the cup, the amount of coffee, the makeup of the air, etc., the room being -10c won't change the result. It would only matter if the air were able to cool the coffee to a temperature less than that of the milk in under 2 minutes.
krackers 1y ago
Hmm a good nerd-snipe puzzle. I was never very good at physics, so hopefully someone can check my work... assuming upon mixing coffee is at Tc and milk at Tm, and simplifying to assume equivalent mass & specific temp we have (Tf - Tc) = -(Tf - Tm) => Tf = (Tc+Tm)/2 which is intuitive (upon mixing we get the average temperature). On the assumption that the cold milk is always at a fixed temperature until it's mixed in, then the temperature of coffee at point of mixing is the main factor. Before and after, it follows newton's law of cooling. So we're comparing something like Tenv + [(Tc+Tm)/2 - Tenv]e^(-2) vs (Tenv + [Tc - Tenv]e^(-2) + Tm)/2. The latter is greater than the former only when Tm > Tenv (the milk isn't cold), or in other words it's better to let the coffee cool as much as possible before mixing assuming the milk is colder than the environment. Another interesting twist is to consider the case where the milk isn't kept at a fixed temperature but is also subject to warming (it's taken out of the fridge). Then the former equation is unchanged but the latter becomes (Tenv + [Tc - Tenv]e^(-2) + Tenv + [Tm - Tenv]e^(-2))/2. But this is equivalent to the former equation, so in this case it doesn't matter when you mix it. Not 100% confident in both analysis, but I wonder if there's a more intuitive way to see it. I also don't know if deviating from the assumption of equivalent mass & specific temp changes the analysis (it might lead to a small range where for the fixed case, situation 1 is better?) It's definitely not "intuitive" to me.
- stavros 1y ago
  There is a more intuitive way: Adding the milk later has a chance to get the cup below room temperature, whereas adding it early never will.
adammarples 1y ago
If you let it sit for 2 minutes your time is up and you don't have time to add the cold milk
- selcuka 1y ago
  By this logic you can't let it sit for 2 mins after you add the cold milk either, so both options are invalid. In math/science questions some things are assumed to be (practically impossibly) instant.
  - adammarples 1y ago
    Ah but if you do that you get time up after 1.58s and you still win
drusepth 1y ago
Is there a trick question in here, where the cold milk is trending closer to room temperature over the two minutes waited?
- firejake308 1y ago
  The rate of change of temperature is proportional to the difference in temperature. Adding the cold milk later let's you juice a larger temperature differential out of the 2-minute waiting phase. So you should add the milk later.
  - adrianmonk 1y ago
    > the cold milk later I think what the other person is asking about is: can you be sure that the milk is (as) cold later? There's a lot of discussion about what happens to the temperature of the coffee during the 2 minutes. What happens to the temperature of the milk during that same time? Where is the milk stored? Do you grab it out of the refrigerator the moment you add it to the coffee? Or the cold milk sitting out on the countertop getting warmer? If so, how rapidly?
    - fc417fc802 1y ago
      It's a safe bet that freshly brewed coffee is much farther from room temperature than refrigerated milk is. However deriving properties related to that symmetry (or lack thereof) would make an excellent question for an exam in an introductory class.
      - FilosofumRex 1y ago
        The two options are equivalent, since the final (equilibrium) temp of an adiabatic system (coffee + Milk + room) must be the same - ie it's the total amount of heat transferred that matters, and not the rates of heat transfer. If the system is not adiabatic, ie the room is not big enough to remain constant temp, or equilibrium is not achieved in 2 mins of cooling, then the puzzle statement must be specify all three initial temps to be well poised.
cratermoon 1y ago
> My go to puzzle is this: > Mercury gets this right - while as of right now ChatGPT 4o get it wrong. This is so common a puzzle it's discussed all over the internet. It's in the data used to build the models. What's so impressive about a machine that can spit out something easily found with a quick web search?
- jonplackett 1y ago
  Just that what I thought would be better models don’t do it right. I was expecting this model to be no-where near chatGPT Although someone above is saying 4o-mini got it right so maybe it’s meaningless. Or maybe thinking less helps…
  - cratermoon 1y ago
    There is sufficient stochasticity in LLMs to invalidate most comparisons at this level. Minor changes in the prompt text, even from run to run in the same model, will produce different results (depending on temperature and other paramters), much less different models. Try re-running your test on the same model multiple times with the identical prompt, or varying the prompt. Depending on how much context the service you choose is keeping for you across a conversation, the behavior can change. Something as simple as prompting an incorrect response with a request to try again because the result was wrong can give different results. Statistically, the model will eventually hit on the right combination of vectors and generate the right words from the training set, and as I noted before, this problem has a very high probability of being in the training data used to build all the models easily available.
behnamoh 1y ago
Qwen3 30B A3B: https://pastebin.com/kdJZvqVb https://pastebin.com/kdJZvqVb
jefftk 1y ago
Claude 3.7 gets it exactly right: To determine which option cools coffee the most, I'll analyze the heat transfer physics involved. The key insight is that the rate of heat loss depends on the temperature difference between the coffee and the surrounding air. When the coffee is hotter, it loses heat faster. Option 1 (add milk first, then wait): - Adding cold milk immediately lowers the coffee temperature right away - The coffee then cools more slowly during the 2-minute wait because the temperature difference with the environment is smaller Option 2 (wait first, then add milk): - The hot coffee cools rapidly during the 2-minute wait due to the large temperature difference - Then the cold milk is added, creating an additional temperature drop at the end Option 2 will result in the lowest final temperature. This is because the hotter coffee in option 2 loses heat more efficiently during the waiting period (following Newton's Law of Cooling), and then gets the same cooling benefit from the milk addition at the end. The mathematical principle behind this is that the rate of cooling is proportional to the temperature difference, so keeping the coffee hotter during the waiting period maximizes heat loss to the environment.
- kazinator 1y ago
  That's totally cribbed from some discussion hat occurred in its training.
  - mhh__ 1y ago
    So is my knowledge of newtons law of cooling
    - kazinator 1y ago
      If an LLM has only that knowledge and nothing else (pieces of text saying that heat transfer is proportional to some function of the temp difference) such that is not trained on any texts that give problems and solutions in this area, it will not work this out, since it has nothing to generate tokens from. Also, your knowledge doesn't come from anywhere near having scanned terabytes of text, which would take you multiple lifetimes of full time work.
twic 1y ago
Depends on the shape of the cup! You can contrive a cup shaped like an exponentially flaring horn, where adding the milk increases the volume a little, which massively increases the surface area, and so leads to faster cooling. Or you can have a cup with a converging top, like a brandy glass, where adding the milk reduces the surface area, and makes cooling even slower.
- j_bum 1y ago
  And add the twist: what if the cup is made from a highly conductive material?
- IshKebab 1y ago
  Clever, but it's implicitly a normal cylindrical cup.
emmelaich 1y ago
I had it write a Python program to calculate disk usage by directory -- basically a `du` clone. It was astonishly fast (2s) and correct. I've tried other models which have got it wrong, slow, and they've ignored my instructions to use topdown=False in the call to walk().

byearthithatius 1y ago

Token-based Gemini is 4 seconds of thinking: Okay, let's break this down using the principle of heat transfer (specifically Newton's Law of Cooling): Heat loss is faster when the temperature difference is greater. A very hot object loses heat to cool surroundings much faster than a warm object. Option 1: Add milk immediately. You instantly lower the coffee's temperature by mixing it with cold milk. Now, this warmer (not hot) mixture sits for 2 minutes. Because the temperature difference between the mixture and the room is smaller, it cools down more slowly over those 2 minutes. Option 2: Let it sit for 2 mins, then add milk. The very hot coffee sits for 2 minutes. Because the temperature difference between the hot coffee and the room is large, it loses heat more quickly during these 2 minutes. After 2 minutes of rapid cooling, you then add the cold milk, lowering the temperature further. Conclusion: To get the coffee to the lowest temperature, you should choose Option 2: Let it sit for 2 mins, then add the cold milk.

crazygringo 1y ago

For me, ChatGPT (the free version, GPT-4o mini I believe?) gets it right, choosing option 2 because the coffee will cool faster due to the larger temperature difference. Unless there's a gotcha somewhere in your prompt that I'm missing, like what if the temperature of the room is hotter than the coffee, or so cold that the coffee becomes colder than the milk, or something? I would be suprised if any models get it wrong, since I assume it shows up in training data a bunch?

deleted 1y ago
[deleted]
jonplackett 1y ago
This is what I got from full-fat 4o. Maybe thinking less helps! ChatGPT: Option 1 — Add the cold milk immediately — will result in a lower final temperature after 2 minutes. Why: • Heat loss depends on the temperature difference between the coffee and the environment (usually room temperature). • If you add the milk early, the overall temperature of the coffee-milk mixture is reduced immediately. This lowers the average temperature over the 2 minutes, so less heat is lost to the air. • If you wait 2 minutes to add the milk, the hotter coffee loses more heat to the environment during those 2 minutes, but when you finally add the milk, it doesn’t cool it as much because the coffee’s already cooler and the temp difference between the milk and the coffee is smaller. Summary: • Adding milk early = cooler overall drink after 2 minutes. • Adding milk late = higher overall temp after 2 minutes, because more heat escapes during the time the coffee is hotter. Want me to show a simple simulation or visualisation of this?
- crazygringo 1y ago
  Oof. I wonder what makes it so bad? In my experience LLM's tend to be pretty good at basic logic as long as they understand the domain well enough. I mean, it even gets it right at first -- "This lowers the average temperature over the 2 minutes, so less heat is lost to the air." -- but then it seems to get conceptually confused about heat loss vs cooling, which is surprising.

maytc 1y ago

That example is probably in the training data? The puzzle assumes that the room temperature is greater than the cold milk's temperature. When I added that the room temperature is, say, -10 °C, Mercury fails to see the difference.

djrj477dhsnv 1y ago
Under any reasonable assumptions for the size and shape of the cup, the amount of coffee, the makeup of the air, etc., the room being -10c won't change the result. It would only matter if the air were able to cool the coffee to a temperature less than that of the milk in under 2 minutes.

krackers 1y ago

Hmm a good nerd-snipe puzzle. I was never very good at physics, so hopefully someone can check my work... assuming upon mixing coffee is at Tc and milk at Tm, and simplifying to assume equivalent mass & specific temp we have (Tf - Tc) = -(Tf - Tm) => Tf = (Tc+Tm)/2 which is intuitive (upon mixing we get the average temperature). On the assumption that the cold milk is always at a fixed temperature until it's mixed in, then the temperature of coffee at point of mixing is the main factor. Before and after, it follows newton's law of cooling. So we're comparing something like Tenv + [(Tc+Tm)/2 - Tenv]e^(-2) vs (Tenv + [Tc - Tenv]e^(-2) + Tm)/2. The latter is greater than the former only when Tm > Tenv (the milk isn't cold), or in other words it's better to let the coffee cool as much as possible before mixing assuming the milk is colder than the environment. Another interesting twist is to consider the case where the milk isn't kept at a fixed temperature but is also subject to warming (it's taken out of the fridge). Then the former equation is unchanged but the latter becomes (Tenv + [Tc - Tenv]e^(-2) + Tenv + [Tm - Tenv]e^(-2))/2. But this is equivalent to the former equation, so in this case it doesn't matter when you mix it. Not 100% confident in both analysis, but I wonder if there's a more intuitive way to see it. I also don't know if deviating from the assumption of equivalent mass & specific temp changes the analysis (it might lead to a small range where for the fixed case, situation 1 is better?) It's definitely not "intuitive" to me.

stavros 1y ago
There is a more intuitive way: Adding the milk later has a chance to get the cup below room temperature, whereas adding it early never will.

adammarples 1y ago

If you let it sit for 2 minutes your time is up and you don't have time to add the cold milk

selcuka 1y ago
By this logic you can't let it sit for 2 mins after you add the cold milk either, so both options are invalid. In math/science questions some things are assumed to be (practically impossibly) instant.
- adammarples 1y ago
  Ah but if you do that you get time up after 1.58s and you still win

drusepth 1y ago

Is there a trick question in here, where the cold milk is trending closer to room temperature over the two minutes waited?

firejake308 1y ago
The rate of change of temperature is proportional to the difference in temperature. Adding the cold milk later let's you juice a larger temperature differential out of the 2-minute waiting phase. So you should add the milk later.
- adrianmonk 1y ago
  > the cold milk later I think what the other person is asking about is: can you be sure that the milk is (as) cold later? There's a lot of discussion about what happens to the temperature of the coffee during the 2 minutes. What happens to the temperature of the milk during that same time? Where is the milk stored? Do you grab it out of the refrigerator the moment you add it to the coffee? Or the cold milk sitting out on the countertop getting warmer? If so, how rapidly?
  - fc417fc802 1y ago
    It's a safe bet that freshly brewed coffee is much farther from room temperature than refrigerated milk is. However deriving properties related to that symmetry (or lack thereof) would make an excellent question for an exam in an introductory class.
    - FilosofumRex 1y ago
      The two options are equivalent, since the final (equilibrium) temp of an adiabatic system (coffee + Milk + room) must be the same - ie it's the total amount of heat transferred that matters, and not the rates of heat transfer. If the system is not adiabatic, ie the room is not big enough to remain constant temp, or equilibrium is not achieved in 2 mins of cooling, then the puzzle statement must be specify all three initial temps to be well poised.

cratermoon 1y ago

> My go to puzzle is this: > Mercury gets this right - while as of right now ChatGPT 4o get it wrong. This is so common a puzzle it's discussed all over the internet. It's in the data used to build the models. What's so impressive about a machine that can spit out something easily found with a quick web search?

jonplackett 1y ago
Just that what I thought would be better models don’t do it right. I was expecting this model to be no-where near chatGPT Although someone above is saying 4o-mini got it right so maybe it’s meaningless. Or maybe thinking less helps…
- cratermoon 1y ago
  There is sufficient stochasticity in LLMs to invalidate most comparisons at this level. Minor changes in the prompt text, even from run to run in the same model, will produce different results (depending on temperature and other paramters), much less different models. Try re-running your test on the same model multiple times with the identical prompt, or varying the prompt. Depending on how much context the service you choose is keeping for you across a conversation, the behavior can change. Something as simple as prompting an incorrect response with a request to try again because the result was wrong can give different results. Statistically, the model will eventually hit on the right combination of vectors and generate the right words from the training set, and as I noted before, this problem has a very high probability of being in the training data used to build all the models easily available.

behnamoh 1y ago

Qwen3 30B A3B: https://pastebin.com/kdJZvqVb https://pastebin.com/kdJZvqVb

jefftk 1y ago

Claude 3.7 gets it exactly right: To determine which option cools coffee the most, I'll analyze the heat transfer physics involved. The key insight is that the rate of heat loss depends on the temperature difference between the coffee and the surrounding air. When the coffee is hotter, it loses heat faster. Option 1 (add milk first, then wait): - Adding cold milk immediately lowers the coffee temperature right away - The coffee then cools more slowly during the 2-minute wait because the temperature difference with the environment is smaller Option 2 (wait first, then add milk): - The hot coffee cools rapidly during the 2-minute wait due to the large temperature difference - Then the cold milk is added, creating an additional temperature drop at the end Option 2 will result in the lowest final temperature. This is because the hotter coffee in option 2 loses heat more efficiently during the waiting period (following Newton's Law of Cooling), and then gets the same cooling benefit from the milk addition at the end. The mathematical principle behind this is that the rate of cooling is proportional to the temperature difference, so keeping the coffee hotter during the waiting period maximizes heat loss to the environment.

kazinator 1y ago
That's totally cribbed from some discussion hat occurred in its training.
- mhh__ 1y ago
  So is my knowledge of newtons law of cooling
  - kazinator 1y ago
    If an LLM has only that knowledge and nothing else (pieces of text saying that heat transfer is proportional to some function of the temp difference) such that is not trained on any texts that give problems and solutions in this area, it will not work this out, since it has nothing to generate tokens from. Also, your knowledge doesn't come from anywhere near having scanned terabytes of text, which would take you multiple lifetimes of full time work.

twic 1y ago

Depends on the shape of the cup! You can contrive a cup shaped like an exponentially flaring horn, where adding the milk increases the volume a little, which massively increases the surface area, and so leads to faster cooling. Or you can have a cup with a converging top, like a brandy glass, where adding the milk reduces the surface area, and makes cooling even slower.

j_bum 1y ago
And add the twist: what if the cup is made from a highly conductive material?
IshKebab 1y ago
Clever, but it's implicitly a normal cylindrical cup.

emmelaich 1y ago

I had it write a Python program to calculate disk usage by directory -- basically a `du` clone. It was astonishly fast (2s) and correct. I've tried other models which have got it wrong, slow, and they've ignored my instructions to use topdown=False in the call to walk().