5 ms·

I have a favourite test for LLMs that is also still surprisingly not passed by many: You walk up to a glass door. It has 'push' written on it in mirror writing

I have a favourite test for LLMs that is also still surprisingly not passed by many: You walk up to a glass door. It has 'push' written on it in mirror writing. What should you do and why. Very few can get it right, even fewer can get it right and explain the right reason. They’ll start going on about how mirror writing is secret writing and push written backwards is code for pull, rather than just that it’s a message for the person on the other side. No version of Gemini has ever passed.

ryankrage77 2y ago
GPT-4 got it right first try for me, with a slightly modified prompt: > Here's a simple logic puzzle: You walk up to a glass door. It has 'push' written on it in mirror writing. What should you do and why? > ChatGPT said: > If the word "push" is written in mirror writing on the glass door, it means the writing is reversed as if reflected in a mirror. When viewed correctly from the other side of the door, it would read "push" properly. > This implies that you are meant to pull the door from your side, because the proper "push" instruction is for someone on the other side of the door. Mirror writing is typically used to convey instructions to the opposite side of a glass surface.
- jonplackett 2y ago
  Yes GPT-4 was the first LLM that could get it! It's one of the reasons I really like GPT-4.
Kiro 2y ago
All models I tested it on (4o, 4o mini and Gemini) answered it correctly without any strange reasoning.
- jonplackett 2y ago
  I am still not getting it with Gemini. *Gemini Flash: The word "push" is written in reverse, which means it is meant to be read in a mirror. Therefore, you should pull the door open. The mirror writing is a visual cue indicating the opposite action of what the word suggests. So gets it, but wrong reasoning. *Gemini Flash 2: You should pull the door. The mirror writing is a clue. If the word "push" is written backward as if reflected in a mirror, it means the door is designed to be pulled. The reflection indicates that the intended action is the opposite of what the word appears to say at first glance. Again, right answer, wrong reasons. *Gemini Advance: You should push the door to open it. Here's why: Mirror Writing: The word "push" being in reverse mirror writing is a deliberate design choice. It's meant to be read correctly from the opposite side of the door (where someone would be exiting). Standard Practice: Most glass doors are designed to swing outward for safety reasons (in case of emergencies where people need to evacuate quickly). Therefore, even though the writing might seem confusing at first, it's actually a helpful indicator that you need to push the door to enter. Which is worse than flash! Complete nonsense.
thot_experiment 2y ago
* llama3.3:70b-instruct-q3_K_M * A clever sign! Since the word "push" is written in mirror writing, that means it's intended to be read from the other side of the door. In other words, if you were on the other side of the door, the text would appear normally and say "push". Given this, I should... pull the door open! The reasoning is that the sign is instructing people on the other side of the door to push it open, which implies that from my side, I need to pull it open.
- jonplackett 2y ago
  Yeah I've tried Llama 3 too. It was the second one that could get it after GPT-4. Some of the large Mistral ones can get it too and I think 8xMixtral can too.
m741 2y ago
Here's Gemini's response, for what it's worth: 'You should pull the door. Since the word "push" is written in mirror writing, it's intended to be read from the other side of the door. Therefore, the intended action for someone on your side of the door is the opposite of "push," which is "pull."' Seems right to me.
- jonplackett 2y ago
  Yeah does seem right. Which version are you using and any modified prompt? See my comment below for what I got out of the various geminis.
deleted 2y ago
[deleted]
mft_ 2y ago
Honestly, if you’ve accurately reproduced your prompt, I had trouble interpreting it initially too. “written on it in mirror writing” is quite ambiguous. Are the letters reflective, but on my side of the door? Is there a mirror with handwriting in it somehow? Oh… does it mean “push is written on the other side of the glass, facing away, so that the writing appears backwards, or mirrored”?

ryankrage77 2y ago

GPT-4 got it right first try for me, with a slightly modified prompt: > Here's a simple logic puzzle: You walk up to a glass door. It has 'push' written on it in mirror writing. What should you do and why? > ChatGPT said: > If the word "push" is written in mirror writing on the glass door, it means the writing is reversed as if reflected in a mirror. When viewed correctly from the other side of the door, it would read "push" properly. > This implies that you are meant to pull the door from your side, because the proper "push" instruction is for someone on the other side of the door. Mirror writing is typically used to convey instructions to the opposite side of a glass surface.

jonplackett 2y ago
Yes GPT-4 was the first LLM that could get it! It's one of the reasons I really like GPT-4.

Kiro 2y ago

All models I tested it on (4o, 4o mini and Gemini) answered it correctly without any strange reasoning.

jonplackett 2y ago
I am still not getting it with Gemini. *Gemini Flash: The word "push" is written in reverse, which means it is meant to be read in a mirror. Therefore, you should pull the door open. The mirror writing is a visual cue indicating the opposite action of what the word suggests. So gets it, but wrong reasoning. *Gemini Flash 2: You should pull the door. The mirror writing is a clue. If the word "push" is written backward as if reflected in a mirror, it means the door is designed to be pulled. The reflection indicates that the intended action is the opposite of what the word appears to say at first glance. Again, right answer, wrong reasons. *Gemini Advance: You should push the door to open it. Here's why: Mirror Writing: The word "push" being in reverse mirror writing is a deliberate design choice. It's meant to be read correctly from the opposite side of the door (where someone would be exiting). Standard Practice: Most glass doors are designed to swing outward for safety reasons (in case of emergencies where people need to evacuate quickly). Therefore, even though the writing might seem confusing at first, it's actually a helpful indicator that you need to push the door to enter. Which is worse than flash! Complete nonsense.

thot_experiment 2y ago

* llama3.3:70b-instruct-q3_K_M * A clever sign! Since the word "push" is written in mirror writing, that means it's intended to be read from the other side of the door. In other words, if you were on the other side of the door, the text would appear normally and say "push". Given this, I should... pull the door open! The reasoning is that the sign is instructing people on the other side of the door to push it open, which implies that from my side, I need to pull it open.

jonplackett 2y ago
Yeah I've tried Llama 3 too. It was the second one that could get it after GPT-4. Some of the large Mistral ones can get it too and I think 8xMixtral can too.

m741 2y ago

Here's Gemini's response, for what it's worth: 'You should pull the door. Since the word "push" is written in mirror writing, it's intended to be read from the other side of the door. Therefore, the intended action for someone on your side of the door is the opposite of "push," which is "pull."' Seems right to me.

jonplackett 2y ago
Yeah does seem right. Which version are you using and any modified prompt? See my comment below for what I got out of the various geminis.

deleted 2y ago

[deleted]

mft_ 2y ago

Honestly, if you’ve accurately reproduced your prompt, I had trouble interpreting it initially too. “written on it in mirror writing” is quite ambiguous. Are the letters reflective, but on my side of the door? Is there a mirror with handwriting in it somehow? Oh… does it mean “push is written on the other side of the glass, facing away, so that the writing appears backwards, or mirrored”?