3 ms·

GPT-4 & Gemini get it right every time. And if I convert it from miles/mph to km/kmh then most of them get it right. Two cars have a 160 km race. Car A drives

GPT-4 & Gemini get it right every time. And if I convert it from miles/mph to km/kmh then most of them get it right. Two cars have a 160 km race. Car A drives 16kmh. Car B drives 8kmh but gets a 80km headstart. Who wins?

xanderlewis 3y ago
I'd be interested to know if they still get it right (i.e. responding with "car A") if it's changed to: "Two cars have a 160km race. Care A drives at 16km/h. Car B drives at 8km/h but gets a 75km head start. Who wins?" (or similar slight perturbations)
- ithkuil 3y ago
  When I tried it with ChatGPT the model understands there is math involved, figures out the formula and spawns an API call to the code execution engine that actually performs the calculation. Results: To determine who wins this race, we'll calculate the time it takes for each car to complete the race using the same formula as before: \[ \text{Time} = \frac{\text{Distance}}{\text{Speed}} \] - Car A starts from 0 km and needs to cover 160 km at a speed of 16 km/h. - Car B starts from 75 km (due to the 75 km head start) and only needs to cover 85 km (since 160 km total distance minus the 75 km head start) at a speed of 8 km/h. Let's calculate the time for each car. Car A will complete the race in 10 hours, while Car B will take 10.625 hours to finish. Therefore, Car A wins the race.
  - xanderlewis 3y ago
    Nice. It’s a good test of (the whole system, including calls to external services, of) ChatGPT itself. I think a pure LLM though would have trouble with these sorts of trick questions…but that’s not surprising.
    - ithkuil 3y ago
      I'm sure the analogy is imperfect but your broca region of the brain is also not good at these sort of trick question yet it can do small talk relatively well even when the rest of the brain has severe dysfunctions
      - xanderlewis 3y ago
        Agreed.
polishdude20 3y ago
I wonder if you gave it a fictional unit of distance and told it so, if it would get it right? Given that a gnorflork is a unit of distance...
- xanderlewis 3y ago
  Good idea. So basically we're seeing if LLMs get this kind of thing: https://math.stackexchange.com/questions/646238/how-do-i-convince-my-students-that-the-choice-of-variable-of-integration-is-irre https://math.stackexchange.com/questions/646238/how-do-i-con... > Draw a graph of the function on the blackboard, showing a and b and a crosshatched area representing the integral. Put an x on the horizontal axis. Erase the x and put a z there. Does that change the area? Erase the z and put a smiley face there. Does the area change? Why/why not?

xanderlewis 3y ago

I'd be interested to know if they still get it right (i.e. responding with "car A") if it's changed to: "Two cars have a 160km race. Care A drives at 16km/h. Car B drives at 8km/h but gets a 75km head start. Who wins?" (or similar slight perturbations)

ithkuil 3y ago
When I tried it with ChatGPT the model understands there is math involved, figures out the formula and spawns an API call to the code execution engine that actually performs the calculation. Results: To determine who wins this race, we'll calculate the time it takes for each car to complete the race using the same formula as before: \[ \text{Time} = \frac{\text{Distance}}{\text{Speed}} \] - Car A starts from 0 km and needs to cover 160 km at a speed of 16 km/h. - Car B starts from 75 km (due to the 75 km head start) and only needs to cover 85 km (since 160 km total distance minus the 75 km head start) at a speed of 8 km/h. Let's calculate the time for each car. Car A will complete the race in 10 hours, while Car B will take 10.625 hours to finish. Therefore, Car A wins the race.
- xanderlewis 3y ago
  Nice. It’s a good test of (the whole system, including calls to external services, of) ChatGPT itself. I think a pure LLM though would have trouble with these sorts of trick questions…but that’s not surprising.
  - ithkuil 3y ago
    I'm sure the analogy is imperfect but your broca region of the brain is also not good at these sort of trick question yet it can do small talk relatively well even when the rest of the brain has severe dysfunctions
    - xanderlewis 3y ago
      Agreed.

polishdude20 3y ago

I wonder if you gave it a fictional unit of distance and told it so, if it would get it right? Given that a gnorflork is a unit of distance...

xanderlewis 3y ago
Good idea. So basically we're seeing if LLMs get this kind of thing: https://math.stackexchange.com/questions/646238/how-do-i-convince-my-students-that-the-choice-of-variable-of-integration-is-irre https://math.stackexchange.com/questions/646238/how-do-i-con... > Draw a graph of the function on the blackboard, showing a and b and a crosshatched area representing the integral. Put an x on the horizontal axis. Erase the x and put a z there. Does that change the area? Erase the z and put a smiley face there. Does the area change? Why/why not?