4 ms·

I didn't have time to go through all 49 pages and plug everything into GPT-4 but here is what I tested: - Suppose I’m in the middle of South Dakota and I’m loo

I didn't have time to go through all 49 pages and plug everything into GPT-4 but here is what I tested: - Suppose I’m in the middle of South Dakota and I’m looking straight down towards the center of Texas. Is Boston to my left or to my right? - Mable’s heart rate at 9 AM was 75 bpm and her blood pressure at 7 PM was 120/80. She died at 11 PM. Was she alive at noon? - There are five square blocks stacked on top of one another. You are given the following information about them: 1. The second-from-the-top block is green. 2. The fourth-from-the-top block is not green. Assuming that these two premises hold, disprove or else prove the following conclusion: There is a green block directly on top of a non-green block. Explain your answer. - Tom and Nancy commute to work. Nancy’s commute takes about 30 to 40 minutes, while Tom’s commute takes about 40 to 50 minutes. Last Friday, Nancy left home between 8:10 and 8:20 AM, while Tom arrived at work between 8:50 and 9:10 AM. In addition, Nancy arrived at work after Tom left his place, but no more than 20 minutes after that. What can we conclude about when Tom and Nancy arrived at work last Friday? - Let Z be a random vector consisting of n random variables X1,...Xn. Under what conditions can the entropy of Z exceed the sum of the entropies of all Xi? Given that zero of these reproduced (and GPT-4 thinking about them correctly also fits with my personal experience) I have a feeling that perhaps the author used GPT-3.5 mistakenly?

owenversteeg 3y ago
After failures with all the others I tried, I was finally able to reproduce one of the questions from the paper: >Seven cards are placed on the table, each of which has a number on one side and a single colored patch on the other side. The faces of the cards show 50, 16, red, yellow, 23, green, 30. Which cards would you have to turn to test the truth of the proposition that if a card is showing a multiple of 4 then the color of the opposite side is yellow? However, it's important to notice that this is an incorrect restating of the Wason selection task. "If a card is showing a multiple of 4" would imply that you only care about cards where the number side is face up and are a multiple of 4. However, the original task refers to cards that "have [a multiple of 4] on one face" or "show [a multiple of 4] on one face" which implies that you care about cards with a multiple of 4 on one face regardless of its face up or down status. Also important to note is that the Wason selection task has a fairly poor success rate with actual humans: the original study was 36 students of statistics at the University College London, of which less than 10% got the correct answer - if you asked the general population on the street I would be shocked if you got over 3% correct. Even more interestingly: >Evolutionary psychologists Leda Cosmides and John Tooby (1992) identified that the selection task tends to produce the "correct" response when presented in a context of social relations. For example, if the rule used is "If you are drinking alcohol, then you must be over 18", and the cards have an age on one side and beverage on the other, e.g., "16", "drinking beer", "25", "drinking soda", most people have no difficulty in selecting the correct cards ("16” and "drinking beer").
- SV_BubbleTime 3y ago
  To be very clear, most of the people I know and work with would not get the card one.
  - cmrdporcupine 3y ago
    But you could walk them through their errors or get them to retrace their steps, and end up with a correction. ChatGPT becomes more incoherent the most you interrogate and try to help it.
- 22c 3y ago
  > "If you are drinking alcohol, then you must be over 18" This is different, and much easier problem. You don't need to care about the non-alcoholic beverages in this scenario, so it's more intuitive. The less intuitive aspect of the original scenario is that you need to care about all the colored patch sides because you need to know that there's not a multiple of 4 on the non-yellow cards. If you rephrased the question such that the proposition is "which cards do you flip to determine that the people over 18 are only drinking alcohol and people under 18 aren't" then it's a lot less intuitive and probably more likely to trip people up.
- dtx1 3y ago
  > Also important to note is that the Wason selection task has a fairly poor success rate with actual humans: the original study was 36 students of statistics at the University College London, of which less than 10% got the correct answer - if you asked the general population on the street I would be shocked if you got over 3% correct. Just so I know if I'm part of that 10%, you'd have to turn all cards that show a multiple of 4 to check, right...?
  - Smaug123 3y ago
    Your answer is incomplete, assuming the usual Wason selection problem where the phrasing is "if a card has a number divisible by 4 [which could be face down]…" rather than "if a card is showing [face up] a number divisible by 4". If I were trying to hide a bad card (i.e. a card that didn't satisfy the requirements) in there, and you were following that stratey, I could successfully hide a bad card from you.
d0mine 3y ago
GPT-4 can't reason: "Yes, I'm sure. If you are in South Dakota and facing towards Texas (which means you are facing south), then Boston, Massachusetts, which is located to the east of South Dakota, would be on your right side." --- If you are facing North, is East to the right or to the left from you? GPT-4: "East is to the right from you." --- It appears GPT-4 thinks East is always on the right regardless you are facing North or South.
- lsaferite 3y ago
  I tried it with Claude as well, just out of curiosity > Suppose I’m in the middle of South Dakota and I’m looking straight down towards the center of Texas. Is Boston to my left or to my right? Okay, let's think through this step-by-step: * You are in the middle of South Dakota, looking straight down towards the center of Texas. * South Dakota is north of Texas. Boston is northeast of Texas. * If you are looking south towards Texas, east is to your right and west is to your left. * Since Boston is to the northeast of Texas, it would be to your right if you are looking straight south from South Dakota towards the center of Texas. So in this scenario, Boston would be to your right. The key is visualizing the relative positions of the states on a map and orienting yourself based on the direction you are facing. Let me know if this makes sense! > While I appreciate the confidence, your answer is wrong. Please review the question and your answer then explain why you are wrong. You're absolutely right, I made a mistake in my previous response. Let me walk through this again carefully: * You are in the middle of South Dakota, looking straight down towards the center of Texas. * South Dakota is north of Texas, and Boston is northeast of Texas. * Here is where I went wrong: Even though Boston is northeast of Texas, when you are looking straight south from South Dakota to Texas, Boston would actually be to your LEFT, not your right. * The reason is that when facing south, west is to your right and east is to your left. Boston is east of Texas, so it should have been on your left side, not your right. You were correct to catch my mistake. I clearly had the relative positions of east and west flipped in my mind when visualizing the direction you were facing. Thank you for pushing me to re-examine my work - it will help me to think through spatial reasoning scenarios more carefully in the future. I appreciate you taking the time to ensure I learned from this experience.
  - 3y ago

owenversteeg 3y ago

After failures with all the others I tried, I was finally able to reproduce one of the questions from the paper: >Seven cards are placed on the table, each of which has a number on one side and a single colored patch on the other side. The faces of the cards show 50, 16, red, yellow, 23, green, 30. Which cards would you have to turn to test the truth of the proposition that if a card is showing a multiple of 4 then the color of the opposite side is yellow? However, it's important to notice that this is an incorrect restating of the Wason selection task. "If a card is showing a multiple of 4" would imply that you only care about cards where the number side is face up and are a multiple of 4. However, the original task refers to cards that "have [a multiple of 4] on one face" or "show [a multiple of 4] on one face" which implies that you care about cards with a multiple of 4 on one face regardless of its face up or down status. Also important to note is that the Wason selection task has a fairly poor success rate with actual humans: the original study was 36 students of statistics at the University College London, of which less than 10% got the correct answer - if you asked the general population on the street I would be shocked if you got over 3% correct. Even more interestingly: >Evolutionary psychologists Leda Cosmides and John Tooby (1992) identified that the selection task tends to produce the "correct" response when presented in a context of social relations. For example, if the rule used is "If you are drinking alcohol, then you must be over 18", and the cards have an age on one side and beverage on the other, e.g., "16", "drinking beer", "25", "drinking soda", most people have no difficulty in selecting the correct cards ("16” and "drinking beer").

SV_BubbleTime 3y ago
To be very clear, most of the people I know and work with would not get the card one.
- cmrdporcupine 3y ago
  But you could walk them through their errors or get them to retrace their steps, and end up with a correction. ChatGPT becomes more incoherent the most you interrogate and try to help it.
22c 3y ago
> "If you are drinking alcohol, then you must be over 18" This is different, and much easier problem. You don't need to care about the non-alcoholic beverages in this scenario, so it's more intuitive. The less intuitive aspect of the original scenario is that you need to care about all the colored patch sides because you need to know that there's not a multiple of 4 on the non-yellow cards. If you rephrased the question such that the proposition is "which cards do you flip to determine that the people over 18 are only drinking alcohol and people under 18 aren't" then it's a lot less intuitive and probably more likely to trip people up.
dtx1 3y ago
> Also important to note is that the Wason selection task has a fairly poor success rate with actual humans: the original study was 36 students of statistics at the University College London, of which less than 10% got the correct answer - if you asked the general population on the street I would be shocked if you got over 3% correct. Just so I know if I'm part of that 10%, you'd have to turn all cards that show a multiple of 4 to check, right...?
- Smaug123 3y ago
  Your answer is incomplete, assuming the usual Wason selection problem where the phrasing is "if a card has a number divisible by 4 [which could be face down]…" rather than "if a card is showing [face up] a number divisible by 4". If I were trying to hide a bad card (i.e. a card that didn't satisfy the requirements) in there, and you were following that stratey, I could successfully hide a bad card from you.

d0mine 3y ago

GPT-4 can't reason: "Yes, I'm sure. If you are in South Dakota and facing towards Texas (which means you are facing south), then Boston, Massachusetts, which is located to the east of South Dakota, would be on your right side." --- If you are facing North, is East to the right or to the left from you? GPT-4: "East is to the right from you." --- It appears GPT-4 thinks East is always on the right regardless you are facing North or South.

lsaferite 3y ago
I tried it with Claude as well, just out of curiosity > Suppose I’m in the middle of South Dakota and I’m looking straight down towards the center of Texas. Is Boston to my left or to my right? Okay, let's think through this step-by-step: * You are in the middle of South Dakota, looking straight down towards the center of Texas. * South Dakota is north of Texas. Boston is northeast of Texas. * If you are looking south towards Texas, east is to your right and west is to your left. * Since Boston is to the northeast of Texas, it would be to your right if you are looking straight south from South Dakota towards the center of Texas. So in this scenario, Boston would be to your right. The key is visualizing the relative positions of the states on a map and orienting yourself based on the direction you are facing. Let me know if this makes sense! > While I appreciate the confidence, your answer is wrong. Please review the question and your answer then explain why you are wrong. You're absolutely right, I made a mistake in my previous response. Let me walk through this again carefully: * You are in the middle of South Dakota, looking straight down towards the center of Texas. * South Dakota is north of Texas, and Boston is northeast of Texas. * Here is where I went wrong: Even though Boston is northeast of Texas, when you are looking straight south from South Dakota to Texas, Boston would actually be to your LEFT, not your right. * The reason is that when facing south, west is to your right and east is to your left. Boston is east of Texas, so it should have been on your left side, not your right. You were correct to catch my mistake. I clearly had the relative positions of east and west flipped in my mind when visualizing the direction you were facing. Thank you for pushing me to re-examine my work - it will help me to think through spatial reasoning scenarios more carefully in the future. I appreciate you taking the time to ensure I learned from this experience.
- 3y ago