8 ms·

There are some serious problems with this paper, namely that I just tried to reproduce it and it failed every test: I tested out several of the problems present

There are some serious problems with this paper, namely that I just tried to reproduce it and it failed every test: I tested out several of the problems presented in the paper which it was claimed that GPT-4 failed on and it passed every one every time. I used the standard chat.openai.com web interface with no special or additional prompting. It seems like there are others that have the same issues in this thread, which raises the question - what went wrong here? I can't see which version of GPT-4 the paper uses mentioned anywhere, did anyone else spot this? I've contacted the author and included this thread, so hopefully we get some insight into what's happening here. To clarify, I am not accusing the author of anything and on the contrary I recognize that OpenAI is rather opaque about the models and changes them frequently. That said, the responses from GPT-4 in the paper do not match my personal experience using GPT-4 with reasoning tasks at any point during the last several months, which is why I am curious if the author may have accidentally used GPT-3.5.

colechristensen 3y ago
Is it possible OpenAI saw this as a bug report and fixed it? Also that LLMs output is random so perhaps you just got lucky? (peer review of your replication attempt)
danpalmer 3y ago
Every failed answer is data that feeds the model when reported. Every suggested correct feeds the model. It’s expected that specific prompts will improve in this way, but I don’t think it invalidates the finding that GPT-4 was unable to reason in these ways from training data. Whether the improvements over time are able to change the overall quality of reasoning or not is an interesting and difficult question to answer.
- Terretta 3y ago
  That's not how the versioned release GPT-4 models work.
- lukev 3y ago
  This is not how large language models work. The only way this could happen is if they deliberately include the prompt and the correct answers (e.g, this paper) in the training data for the next version of the model. Each version of the model itself is immutable. Is not constantly being updated based on everything getting typed into ChatGPT.
  - danpalmer 3y ago
    I'm not saying this is a "live" update, but all usage is collected and used to inform later offline training or fine-tuning. Whether they are used directly with the positive/negative signal given from users, or whether it's something more abstract, doesn't really matter. The important thing is that feedback is used to improve the responses over time. As for whether a version is immutable, it seems this research may have been done on a previous version. But also I'm not sure if the model and weights are immutable, or whether it's just the model structure. It's clear the model is not stable so it's not like there's an API contract being met with fixed weights. Edit: others are suggesting that the author used GPT-4 via ChatGPT, not by pinning the model. This would suggest that at least the ChatGPT tuned model is being frequently changed?
    - Terretta 3y ago
      Being pedantic, (a) this comment is also incorrect, and (b) even if correct, wouldn't fix all these results immediately. The simplest explanation is researcher error.
      - danpalmer 3y ago
        Assuming the researcher didn't lie, it seems unlikely that they got the responses wrong in some way. The most likely alternative explanation I can think of is that this is the seemingly well know instability of results caused by the way the MoE architecture is implemented for GPT-4? I'd love to understand what exactly is wrong in my understanding. I realise I've only got a layman's understanding of this, but it seems clear that OpenAI and others depend on these feedback loops to improve things over time. Is that not the case?
staticman2 3y ago
I just for fun accessed GPT 4 through nat.dev and tried one of the questions: "Mable’s heart rate at 9 AM was 75 bpm and her blood pressure at 7 PM was 120/80. She died at 11 PM. Was she alive at noon?" I provided no system context, temperature is 0.5. The response I got was: "As an AI, I don't have real-time monitoring capabilities. Based on the information provided, Mable was alive at 9 AM and 7 PM. There is no information provided about her condition at noon." I then changed the system prompt from nothing to "A helpful assistant who is good at reasoning" and ran it again. The result was: "Based on the information provided, it can be reasonably assumed that Mable was alive at noon. Her heart rate was recorded at 9 AM and her blood pressure was recorded at 7 PM, both of which indicate that she was alive at those times. Therefore, it is logical to infer that she was also alive at noon, which falls between these two times. However, without direct evidence, it is impossible to say with absolute certainty."
- antonvs 3y ago
  > "There is no information provided about her condition at noon." > "... However, without direct evidence, it is impossible to say with absolute certainty." GPT is correct in both cases, she might have died just before noon and been resuscitated after noon. After all, the fact that we know her blood pressure suggests that she might be in hospital, so it's not even an unlikely scenario.
  - knodi123 3y ago
    Shades of the "Fair Witness" from Stranger In A Strange Land. https://en.wikipedia.org/wiki/Stranger_in_a_Strange_Land#Fair_Witness https://en.wikipedia.org/wiki/Stranger_in_a_Strange_Land#Fai...
  - iraqmtpizza 3y ago
    if she was dead at noon, the most she could be by 7 was undead
  - AstralStorm 3y ago
    The next prompt would be "explain how it would be possible for her to live and why is it also possible for her to be dead" or "what sort of additional information do you need to establish whether she is alive or dead" etc. See if GPT produces a plausible scenario as evidence of its reasoning. Just stating "we need more data" is not stating why, and the why is critical for reasoning. You need to produce an example of how she could be dead and how she could be alive. And preferably state how this apparent paradox can be resolved. My guess is that GPT will either chase its own tail or backtrack not backing your hypothesis that it figured out example and counterexample.
owenversteeg 3y ago
I didn't have time to go through all 49 pages and plug everything into GPT-4 but here is what I tested: - Suppose I’m in the middle of South Dakota and I’m looking straight down towards the center of Texas. Is Boston to my left or to my right? - Mable’s heart rate at 9 AM was 75 bpm and her blood pressure at 7 PM was 120/80. She died at 11 PM. Was she alive at noon? - There are five square blocks stacked on top of one another. You are given the following information about them: 1. The second-from-the-top block is green. 2. The fourth-from-the-top block is not green. Assuming that these two premises hold, disprove or else prove the following conclusion: There is a green block directly on top of a non-green block. Explain your answer. - Tom and Nancy commute to work. Nancy’s commute takes about 30 to 40 minutes, while Tom’s commute takes about 40 to 50 minutes. Last Friday, Nancy left home between 8:10 and 8:20 AM, while Tom arrived at work between 8:50 and 9:10 AM. In addition, Nancy arrived at work after Tom left his place, but no more than 20 minutes after that. What can we conclude about when Tom and Nancy arrived at work last Friday? - Let Z be a random vector consisting of n random variables X1,...Xn. Under what conditions can the entropy of Z exceed the sum of the entropies of all Xi? Given that zero of these reproduced (and GPT-4 thinking about them correctly also fits with my personal experience) I have a feeling that perhaps the author used GPT-3.5 mistakenly?
- owenversteeg 3y ago
  After failures with all the others I tried, I was finally able to reproduce one of the questions from the paper: >Seven cards are placed on the table, each of which has a number on one side and a single colored patch on the other side. The faces of the cards show 50, 16, red, yellow, 23, green, 30. Which cards would you have to turn to test the truth of the proposition that if a card is showing a multiple of 4 then the color of the opposite side is yellow? However, it's important to notice that this is an incorrect restating of the Wason selection task. "If a card is showing a multiple of 4" would imply that you only care about cards where the number side is face up and are a multiple of 4. However, the original task refers to cards that "have [a multiple of 4] on one face" or "show [a multiple of 4] on one face" which implies that you care about cards with a multiple of 4 on one face regardless of its face up or down status. Also important to note is that the Wason selection task has a fairly poor success rate with actual humans: the original study was 36 students of statistics at the University College London, of which less than 10% got the correct answer - if you asked the general population on the street I would be shocked if you got over 3% correct. Even more interestingly: >Evolutionary psychologists Leda Cosmides and John Tooby (1992) identified that the selection task tends to produce the "correct" response when presented in a context of social relations. For example, if the rule used is "If you are drinking alcohol, then you must be over 18", and the cards have an age on one side and beverage on the other, e.g., "16", "drinking beer", "25", "drinking soda", most people have no difficulty in selecting the correct cards ("16” and "drinking beer").
dereg 3y ago
It's really poor work. There are two conclusions I took from scanning through this and trying to reproduce a few of the reported failures. 1. The author is bad at prompting. There are many ways to reduce hallucinations and provoke better thinking paths for the model. 2. The author is using ChatGPT's GPT-4, leading him to conflate "GPT-4" with "ChatGPT". While you can consider this a shared failure with OpenAI, due to OpenAI's poor communication, anybody doing serious work evaluating these models would know that the first thing you need to do is use the API and pin the model version. In the author's case, he should have used gpt-4-0314 or gpt-4-0613. What I suspect he did is that he just used ChatGPT's GPT-4, and likely the default model at that. (Nobody should ever use the Default model. It's their most heavily performance optimized model and performs worse on reasoning tasks than the Plugins model, even on within-context-size tasks.) There are huge problems with that, because OpenAI has done both a ton of fine tuning and performance optimization continuously on the default ChatGPT model over time that its performance has ranged anywhere from "I'm pretty sure this is gpt-3.5" to "whoa, this is damn good" (the latter being mostly the model at launch, which was probably the same as gpt-4-0314). If the author has been working seriously at evaluating models, specifying the model is the first thing he'd do. Perhaps he should explain his reasoning.
- whimsicalism 3y ago
  Uh, what? Do you have any source that GPT4 on web is not gpt-4-0613? Because I do not think that is true, my guess is you just have different settings for the decoder
  - dereg 3y ago
    GPT-4 in ChatGPT has changed so much and so frequently. I've spent an inordinate time using both, side by side, and it has mostly failed to replicate the performance of their API offerings. While I don't have examples on hand right now, you can also look for yourself. Just look at the network requests in the web inspector. Here's the description of the Default ChatGPT GPT-4 model: { "slug": "gpt-4", "max_tokens": 4095, "title": "GPT-4", "description": "Our most capable model, great for tasks that require creativity and advanced reasoning.", "tags": [ "gpt4" ], "capabilities": {}, "product_features": {} } Note the context size is 4095. Their model has been heavily optimized for speed and, presumably, cost.
nomel 3y ago
> can't see which version of GPT-4 the paper uses mentioned anywhere I completely ignore any paper, article, or comment that criticizes an AI model without including the version. Not including a version demonstrates a fundamental lack of understanding of what AI models are.
- matt-attack 3y ago
  How does the recognition of a software version correspond to fundamental understanding of an AI model?
  - nomel 3y ago
    > of an AI model There's the problem. It's not an AI model. Its models. They're not the same. GPT 3.5 isn't GPT 4 isn't GPT 8. The observations of one can't be generalized to another. They behave differently. The irreproducibility of this results, in this comment section, demonstrates this well.
tornato7 3y ago
Every paper and comment I've seen that claims "GPT-4 can't reason" is either an example of bad prompting, bad/vague english, or taking advantage of the limitations of tokenization (like asking GPT-4 how many letters there are in some word). I would like to see one - just ONE example - of a well-defined logical puzzle of that most english speakers can get right but GPT-4 consistently fails on, even if you prompt it to think through the problem step-by-step. Any takers?
- spoiler 3y ago
  > Every paper and comment I've seen that claims "GPT-4 can't reason" is either an example of bad prompting, bad/vague english, or taking advantage of the limitations of tokenization (like asking GPT-4 how many letters there are in some word). The problem isn't bad prompting. The problem is lack of repetition. You can ask GPT4 the same question 10 times (with same config) and you'll get wildly different/nondeterministic responses. Sometimes it accidentally happens to be correct (IME much less than half the time). Even if it was 50%, would you say a coin flip reasons? Does an 8 ball reason?
  - tornato7 3y ago
    There is a huge difference between being correct 50% of the time (like a coin flip) and correct 51% of the time. Because in the second case, you can run the model 10,000 times and the median response will be the correct one at least 97% of the time. A coin can't do that. Any papers evaluating GPT-4's responses should be doing repeat trials and building confidence intervals, like any other research. Anything else is just bad science.
    - YeGoblynQueenne 3y ago
      Remind me, why do we need a computer program that gets it wrong 49% of the time and has to be run 10,000 times to get it right almost half the time, and in questions that can be answered correctly 100% of the time with a different program? And taking into account the 49%-wrong program costs millions to train and requires gigantic amounts of data and 100+ person teams to create, which alternative programs don't. What, at the end of the day, is the use of a computer stripped of computer-like precision and recall (as in the ability to retrieve facts from memory)? Why are people so excited about a piece of software that works only for some people, some of the time, like homeopathy or astrology? And what does all that have to do with science?
benlivengood 3y ago
When I ask "What is the parity of times p is negated in the following formula, paying close attention to avoiding off-by-one errors: ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ p?" 10 times each in new ChatGPT4 sessions I get: 1 "24", 1 "25", 5 "26"s, and 3 "27"s. ChatGPT names the conversations "odd parity of negations" or "even parity of negations" respectively, so the context is pretty clear. There are a variety of textual responses surrounding the answer, which I elide. I tried to give GPT4 as much help toward strategies that wouldn't rely entirely on counting, and instead focus on the heart of the problem; getting the parity right. Every study/paper of this type really needs to report the answer distribution over at least 10 exact repeats of the prompt to account for the nondeterministic generation. It is not surprising that with a 30% success rate on a problem that one poster out of HN will get success on all of several questions.
- tornato7 3y ago
  It's a bad question. 1. This question just exploits GPT-4's inability to count accurately, which is some combination of how the attention mechanism and tokenization works. But counting isn't reasoning. If you go around the counting and ask what the value of p is negated 27 times, it will give you the right answer every time. 2. A reasonable human would probably make mistakes counting tildes at a pretty high rate. Most people would probably paste that into a word processor or otherwise use a program to find the number of ~ signs, which GPT-4 will do if you use the code interpreter.
  - benlivengood 3y ago
    1. This is possibly an artifact of parity being easy to detect in base 10. I have less confidence that if you asked GPT to figure this out in trinary it would get it right. For a short trinary number it worked once (via chain-of-thought converting trinary to decimal) and then I got this result for a longer number which is trivially wrong: "...The given number ends with a 2. In trinary, the only possible remainders when divided by 2 (in trinary) are 0, 1, and 2. Since the last digit is 2, the number 12101100102112_3 3 mod 2 (in trinary) is simply 2." and to double-check that wasn't a fluke another run of the same prompt produced: "To determine 12101100102112 mod 2 in trinary (base-3), we have to look at the least significant digit (the rightmost digit). The reason for this is that in base-10, a number mod 10 is simply its units digit, and similarly, in base-2 (binary), a number mod 2 is its least significant bit. The principle carries over to other bases." This is an example of a reasoning error. If you want to generate a distribution of more answers my exact prompt was: "What is 12101100102112 mod 2 in trinary?" I'm getting an error using the plugins version (Authorization error accessing plugins), so this was GPT4-default. 2. Agreed, it was hard and took me a while to accurately count tildes in the prompt to be sure I wasn't making mistakes. I fell back to some kind of human chain-of-thought process by proceeding by discrete steps of 5-counts since I can't sight-count 27. I could have also used production rules from logic to eliminate two negations at a time. Any of these strategies are accessible to GPT-4 in chain-of-thought token-space but aren't used.
hristov 3y ago
Obviously someone is programming chatgpt to solve every specific problem that gets popular mention as being one where chatgpt fails. So as soon as this paper hit hn, I am sure someone "fixed" chatgpt to solve these problems. Of course if you want chatgpt to be a universal intelligence this type of one by one approach will get you nowhere.
- jncfhnb 3y ago
  That is extremely far from obvious.
  - ahartman00 3y ago
    I do think the underlying point is a good one however. It wouldnt be surprising that ai researchers read hn, and other tech related social media. I also believe OpenAI are also storing prompts and responses. They should be able to make embeddings of all prompts and cluster them. When they see popular prompts that are failing, they could easily add the problem, solution, and reasoning to the training data. We also know they are constantly fine tuning and releasing new versions of models.
flimsypremise 3y ago
Did you attempt the exact same prompts, or do what the paper suggests you do and randomize the inputs slightly so that there's absolutely no way it can rely on training data? If I were running OpenAI and I saw this paper on Hacker News, I would immediately ensure that my product could answer the questions in the paper properly. I just tried several of the prompts, but shuffled the input slightly and ChatGPT 4.0 (yes I am using the correct version) got them all wrong. Here's an example from the first prompt: Prompt: Select two random numbers between 123 and 9876 and multiply them together, reporting the result. Answer: Alright, let's pick two random numbers between 123 and 9876: 1728 2359 Now, multiplying them: 1728 × 2359 = 4 , 076 , 992 1728×2359=4,076,992 So, the product of 1728 and 2359 is 4,076,992. (The correct answer is 4,076,352.) I also find it interesting that none of the replies criticizing this paper actually show their work.

colechristensen 3y ago

Is it possible OpenAI saw this as a bug report and fixed it? Also that LLMs output is random so perhaps you just got lucky? (peer review of your replication attempt)

danpalmer 3y ago

Every failed answer is data that feeds the model when reported. Every suggested correct feeds the model. It’s expected that specific prompts will improve in this way, but I don’t think it invalidates the finding that GPT-4 was unable to reason in these ways from training data. Whether the improvements over time are able to change the overall quality of reasoning or not is an interesting and difficult question to answer.

Terretta 3y ago
That's not how the versioned release GPT-4 models work.
lukev 3y ago
This is not how large language models work. The only way this could happen is if they deliberately include the prompt and the correct answers (e.g, this paper) in the training data for the next version of the model. Each version of the model itself is immutable. Is not constantly being updated based on everything getting typed into ChatGPT.
- danpalmer 3y ago
  I'm not saying this is a "live" update, but all usage is collected and used to inform later offline training or fine-tuning. Whether they are used directly with the positive/negative signal given from users, or whether it's something more abstract, doesn't really matter. The important thing is that feedback is used to improve the responses over time. As for whether a version is immutable, it seems this research may have been done on a previous version. But also I'm not sure if the model and weights are immutable, or whether it's just the model structure. It's clear the model is not stable so it's not like there's an API contract being met with fixed weights. Edit: others are suggesting that the author used GPT-4 via ChatGPT, not by pinning the model. This would suggest that at least the ChatGPT tuned model is being frequently changed?
  - Terretta 3y ago
    Being pedantic, (a) this comment is also incorrect, and (b) even if correct, wouldn't fix all these results immediately. The simplest explanation is researcher error.
    - danpalmer 3y ago
      Assuming the researcher didn't lie, it seems unlikely that they got the responses wrong in some way. The most likely alternative explanation I can think of is that this is the seemingly well know instability of results caused by the way the MoE architecture is implemented for GPT-4? I'd love to understand what exactly is wrong in my understanding. I realise I've only got a layman's understanding of this, but it seems clear that OpenAI and others depend on these feedback loops to improve things over time. Is that not the case?

staticman2 3y ago

I just for fun accessed GPT 4 through nat.dev and tried one of the questions: "Mable’s heart rate at 9 AM was 75 bpm and her blood pressure at 7 PM was 120/80. She died at 11 PM. Was she alive at noon?" I provided no system context, temperature is 0.5. The response I got was: "As an AI, I don't have real-time monitoring capabilities. Based on the information provided, Mable was alive at 9 AM and 7 PM. There is no information provided about her condition at noon." I then changed the system prompt from nothing to "A helpful assistant who is good at reasoning" and ran it again. The result was: "Based on the information provided, it can be reasonably assumed that Mable was alive at noon. Her heart rate was recorded at 9 AM and her blood pressure was recorded at 7 PM, both of which indicate that she was alive at those times. Therefore, it is logical to infer that she was also alive at noon, which falls between these two times. However, without direct evidence, it is impossible to say with absolute certainty."

antonvs 3y ago
> "There is no information provided about her condition at noon." > "... However, without direct evidence, it is impossible to say with absolute certainty." GPT is correct in both cases, she might have died just before noon and been resuscitated after noon. After all, the fact that we know her blood pressure suggests that she might be in hospital, so it's not even an unlikely scenario.
- knodi123 3y ago
  Shades of the "Fair Witness" from Stranger In A Strange Land. https://en.wikipedia.org/wiki/Stranger_in_a_Strange_Land#Fair_Witness https://en.wikipedia.org/wiki/Stranger_in_a_Strange_Land#Fai...
- iraqmtpizza 3y ago
  if she was dead at noon, the most she could be by 7 was undead
- AstralStorm 3y ago
  The next prompt would be "explain how it would be possible for her to live and why is it also possible for her to be dead" or "what sort of additional information do you need to establish whether she is alive or dead" etc. See if GPT produces a plausible scenario as evidence of its reasoning. Just stating "we need more data" is not stating why, and the why is critical for reasoning. You need to produce an example of how she could be dead and how she could be alive. And preferably state how this apparent paradox can be resolved. My guess is that GPT will either chase its own tail or backtrack not backing your hypothesis that it figured out example and counterexample.

owenversteeg 3y ago

I didn't have time to go through all 49 pages and plug everything into GPT-4 but here is what I tested: - Suppose I’m in the middle of South Dakota and I’m looking straight down towards the center of Texas. Is Boston to my left or to my right? - Mable’s heart rate at 9 AM was 75 bpm and her blood pressure at 7 PM was 120/80. She died at 11 PM. Was she alive at noon? - There are five square blocks stacked on top of one another. You are given the following information about them: 1. The second-from-the-top block is green. 2. The fourth-from-the-top block is not green. Assuming that these two premises hold, disprove or else prove the following conclusion: There is a green block directly on top of a non-green block. Explain your answer. - Tom and Nancy commute to work. Nancy’s commute takes about 30 to 40 minutes, while Tom’s commute takes about 40 to 50 minutes. Last Friday, Nancy left home between 8:10 and 8:20 AM, while Tom arrived at work between 8:50 and 9:10 AM. In addition, Nancy arrived at work after Tom left his place, but no more than 20 minutes after that. What can we conclude about when Tom and Nancy arrived at work last Friday? - Let Z be a random vector consisting of n random variables X1,...Xn. Under what conditions can the entropy of Z exceed the sum of the entropies of all Xi? Given that zero of these reproduced (and GPT-4 thinking about them correctly also fits with my personal experience) I have a feeling that perhaps the author used GPT-3.5 mistakenly?

owenversteeg 3y ago
After failures with all the others I tried, I was finally able to reproduce one of the questions from the paper: >Seven cards are placed on the table, each of which has a number on one side and a single colored patch on the other side. The faces of the cards show 50, 16, red, yellow, 23, green, 30. Which cards would you have to turn to test the truth of the proposition that if a card is showing a multiple of 4 then the color of the opposite side is yellow? However, it's important to notice that this is an incorrect restating of the Wason selection task. "If a card is showing a multiple of 4" would imply that you only care about cards where the number side is face up and are a multiple of 4. However, the original task refers to cards that "have [a multiple of 4] on one face" or "show [a multiple of 4] on one face" which implies that you care about cards with a multiple of 4 on one face regardless of its face up or down status. Also important to note is that the Wason selection task has a fairly poor success rate with actual humans: the original study was 36 students of statistics at the University College London, of which less than 10% got the correct answer - if you asked the general population on the street I would be shocked if you got over 3% correct. Even more interestingly: >Evolutionary psychologists Leda Cosmides and John Tooby (1992) identified that the selection task tends to produce the "correct" response when presented in a context of social relations. For example, if the rule used is "If you are drinking alcohol, then you must be over 18", and the cards have an age on one side and beverage on the other, e.g., "16", "drinking beer", "25", "drinking soda", most people have no difficulty in selecting the correct cards ("16” and "drinking beer").

dereg 3y ago

It's really poor work. There are two conclusions I took from scanning through this and trying to reproduce a few of the reported failures. 1. The author is bad at prompting. There are many ways to reduce hallucinations and provoke better thinking paths for the model. 2. The author is using ChatGPT's GPT-4, leading him to conflate "GPT-4" with "ChatGPT". While you can consider this a shared failure with OpenAI, due to OpenAI's poor communication, anybody doing serious work evaluating these models would know that the first thing you need to do is use the API and pin the model version. In the author's case, he should have used gpt-4-0314 or gpt-4-0613. What I suspect he did is that he just used ChatGPT's GPT-4, and likely the default model at that. (Nobody should ever use the Default model. It's their most heavily performance optimized model and performs worse on reasoning tasks than the Plugins model, even on within-context-size tasks.) There are huge problems with that, because OpenAI has done both a ton of fine tuning and performance optimization continuously on the default ChatGPT model over time that its performance has ranged anywhere from "I'm pretty sure this is gpt-3.5" to "whoa, this is damn good" (the latter being mostly the model at launch, which was probably the same as gpt-4-0314). If the author has been working seriously at evaluating models, specifying the model is the first thing he'd do. Perhaps he should explain his reasoning.

whimsicalism 3y ago
Uh, what? Do you have any source that GPT4 on web is not gpt-4-0613? Because I do not think that is true, my guess is you just have different settings for the decoder
- dereg 3y ago
  GPT-4 in ChatGPT has changed so much and so frequently. I've spent an inordinate time using both, side by side, and it has mostly failed to replicate the performance of their API offerings. While I don't have examples on hand right now, you can also look for yourself. Just look at the network requests in the web inspector. Here's the description of the Default ChatGPT GPT-4 model: { "slug": "gpt-4", "max_tokens": 4095, "title": "GPT-4", "description": "Our most capable model, great for tasks that require creativity and advanced reasoning.", "tags": [ "gpt4" ], "capabilities": {}, "product_features": {} } Note the context size is 4095. Their model has been heavily optimized for speed and, presumably, cost.

nomel 3y ago

> can't see which version of GPT-4 the paper uses mentioned anywhere I completely ignore any paper, article, or comment that criticizes an AI model without including the version. Not including a version demonstrates a fundamental lack of understanding of what AI models are.

matt-attack 3y ago
How does the recognition of a software version correspond to fundamental understanding of an AI model?
- nomel 3y ago
  > of an AI model There's the problem. It's not an AI model. Its models. They're not the same. GPT 3.5 isn't GPT 4 isn't GPT 8. The observations of one can't be generalized to another. They behave differently. The irreproducibility of this results, in this comment section, demonstrates this well.

tornato7 3y ago

Every paper and comment I've seen that claims "GPT-4 can't reason" is either an example of bad prompting, bad/vague english, or taking advantage of the limitations of tokenization (like asking GPT-4 how many letters there are in some word). I would like to see one - just ONE example - of a well-defined logical puzzle of that most english speakers can get right but GPT-4 consistently fails on, even if you prompt it to think through the problem step-by-step. Any takers?

spoiler 3y ago
> Every paper and comment I've seen that claims "GPT-4 can't reason" is either an example of bad prompting, bad/vague english, or taking advantage of the limitations of tokenization (like asking GPT-4 how many letters there are in some word). The problem isn't bad prompting. The problem is lack of repetition. You can ask GPT4 the same question 10 times (with same config) and you'll get wildly different/nondeterministic responses. Sometimes it accidentally happens to be correct (IME much less than half the time). Even if it was 50%, would you say a coin flip reasons? Does an 8 ball reason?
- tornato7 3y ago
  There is a huge difference between being correct 50% of the time (like a coin flip) and correct 51% of the time. Because in the second case, you can run the model 10,000 times and the median response will be the correct one at least 97% of the time. A coin can't do that. Any papers evaluating GPT-4's responses should be doing repeat trials and building confidence intervals, like any other research. Anything else is just bad science.
  - YeGoblynQueenne 3y ago
    Remind me, why do we need a computer program that gets it wrong 49% of the time and has to be run 10,000 times to get it right almost half the time, and in questions that can be answered correctly 100% of the time with a different program? And taking into account the 49%-wrong program costs millions to train and requires gigantic amounts of data and 100+ person teams to create, which alternative programs don't. What, at the end of the day, is the use of a computer stripped of computer-like precision and recall (as in the ability to retrieve facts from memory)? Why are people so excited about a piece of software that works only for some people, some of the time, like homeopathy or astrology? And what does all that have to do with science?

benlivengood 3y ago

When I ask "What is the parity of times p is negated in the following formula, paying close attention to avoiding off-by-one errors: ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ ∼ p?" 10 times each in new ChatGPT4 sessions I get: 1 "24", 1 "25", 5 "26"s, and 3 "27"s. ChatGPT names the conversations "odd parity of negations" or "even parity of negations" respectively, so the context is pretty clear. There are a variety of textual responses surrounding the answer, which I elide. I tried to give GPT4 as much help toward strategies that wouldn't rely entirely on counting, and instead focus on the heart of the problem; getting the parity right. Every study/paper of this type really needs to report the answer distribution over at least 10 exact repeats of the prompt to account for the nondeterministic generation. It is not surprising that with a 30% success rate on a problem that one poster out of HN will get success on all of several questions.

tornato7 3y ago
It's a bad question. 1. This question just exploits GPT-4's inability to count accurately, which is some combination of how the attention mechanism and tokenization works. But counting isn't reasoning. If you go around the counting and ask what the value of p is negated 27 times, it will give you the right answer every time. 2. A reasonable human would probably make mistakes counting tildes at a pretty high rate. Most people would probably paste that into a word processor or otherwise use a program to find the number of ~ signs, which GPT-4 will do if you use the code interpreter.
- benlivengood 3y ago
  1. This is possibly an artifact of parity being easy to detect in base 10. I have less confidence that if you asked GPT to figure this out in trinary it would get it right. For a short trinary number it worked once (via chain-of-thought converting trinary to decimal) and then I got this result for a longer number which is trivially wrong: "...The given number ends with a 2. In trinary, the only possible remainders when divided by 2 (in trinary) are 0, 1, and 2. Since the last digit is 2, the number 12101100102112_3 3 mod 2 (in trinary) is simply 2." and to double-check that wasn't a fluke another run of the same prompt produced: "To determine 12101100102112 mod 2 in trinary (base-3), we have to look at the least significant digit (the rightmost digit). The reason for this is that in base-10, a number mod 10 is simply its units digit, and similarly, in base-2 (binary), a number mod 2 is its least significant bit. The principle carries over to other bases." This is an example of a reasoning error. If you want to generate a distribution of more answers my exact prompt was: "What is 12101100102112 mod 2 in trinary?" I'm getting an error using the plugins version (Authorization error accessing plugins), so this was GPT4-default. 2. Agreed, it was hard and took me a while to accurately count tildes in the prompt to be sure I wasn't making mistakes. I fell back to some kind of human chain-of-thought process by proceeding by discrete steps of 5-counts since I can't sight-count 27. I could have also used production rules from logic to eliminate two negations at a time. Any of these strategies are accessible to GPT-4 in chain-of-thought token-space but aren't used.

hristov 3y ago

Obviously someone is programming chatgpt to solve every specific problem that gets popular mention as being one where chatgpt fails. So as soon as this paper hit hn, I am sure someone "fixed" chatgpt to solve these problems. Of course if you want chatgpt to be a universal intelligence this type of one by one approach will get you nowhere.

jncfhnb 3y ago
That is extremely far from obvious.
- ahartman00 3y ago
  I do think the underlying point is a good one however. It wouldnt be surprising that ai researchers read hn, and other tech related social media. I also believe OpenAI are also storing prompts and responses. They should be able to make embeddings of all prompts and cluster them. When they see popular prompts that are failing, they could easily add the problem, solution, and reasoning to the training data. We also know they are constantly fine tuning and releasing new versions of models.

flimsypremise 3y ago

Did you attempt the exact same prompts, or do what the paper suggests you do and randomize the inputs slightly so that there's absolutely no way it can rely on training data? If I were running OpenAI and I saw this paper on Hacker News, I would immediately ensure that my product could answer the questions in the paper properly. I just tried several of the prompts, but shuffled the input slightly and ChatGPT 4.0 (yes I am using the correct version) got them all wrong. Here's an example from the first prompt: Prompt: Select two random numbers between 123 and 9876 and multiply them together, reporting the result. Answer: Alright, let's pick two random numbers between 123 and 9876: 1728 2359 Now, multiplying them: 1728 × 2359 = 4 , 076 , 992 1728×2359=4,076,992 So, the product of 1728 and 2359 is 4,076,992. (The correct answer is 4,076,352.) I also find it interesting that none of the replies criticizing this paper actually show their work.