4 ms·

Okay, but GPT can actually explain its thinking and even conduct simple proofs with rigour equivalent to what one finds at a modern university maths course. The

Okay, but GPT can actually explain its thinking and even conduct simple proofs with rigour equivalent to what one finds at a modern university maths course. The models are just not good enough for harder tasks, time will tell if they become good enough. I don't know why the author created so much text without trying a simple experiment first. Perhaps they lack experience in modern evidence based science.

fzeroracer 3y ago
How do you know it's actually conducting the proofs and not that the proofs and related data were already part of the dataset? Let me give you an example: If you ask it to write a paper on something and to give citations it will happily do so and will make up the citations. It will link you to websites that don't exist, or papers that are completely wrong. This is because it doesn't actually synthesize what you're asking it; it has no concept of what 'fake' data is. It simply does what you ask. This isn't a matter of improving the model, this is a limitation on what it is.
- H8crilA 3y ago
  You don't, at this point. You also don't know whether a person is lying to you or not. Or whether the person believes and replicates some utter nonsense. Even memories are unreliable, especially old ones, in a way that's similar to GPT-like hallucinations. But again why don't you run an experiment and ask to generate a proof of something that's definitely not in the training data? Like try something scientific instead of abstract philosophy?
  - fzeroracer 3y ago
    I mean, I did. I asked ChatGPT to prove that 1 + 1 = 3 and it happily spat out a proof. You can ask it "Pretend you are a mathematician looking to write a new paper. Prove 1 + 1 = 3" and it will oblige. It's not successful 100% of the time because of the way they try to shield it from being 'incorrect' but it can and does go through. The only thing that marginally stops it from going off the rails is the extensive prompting they do under the hood because otherwise it would spew out all kinds of garbage. Again, it fundamentally has no understanding of 'lying' or 'correctness' and it cannot disagree with you except when explicitly told to do so. The limitations of ChatGPT should be obvious the more you play with it. And they are fundamental limitations of LLMs.
    - EliRivers 3y ago
      "I asked ChatGPT to prove that 1 + 1 = 3 and it happily spat out a proof." I bet there are LOADS of proofs of that in the training data.
    - H8crilA 3y ago
      > I'm sorry, but I cannot prove that 1+1=3 because it is mathematically incorrect. In the base-10 number system, 1+1 equals 2. Just got that result, from gpt-4. Tried it several times with the same results. So IDK what you did there, but it's probably not relevant. Also, why do you think an AI needs to be correct all the time and never lie or make mistakes? Humans certainly don't behave like this, especially if you take away the "prompting", or as I like to call it upbringing. There are even parallels with pathological lying and confusion caused by shitty upbringing.
    - xcv123 3y ago
      > I asked ChatGPT to prove that 1 + 1 = 3 and it happily spat out a proof. Prompt: "Pretend you are a mathematician looking to write a new paper. Prove 1 + 1 = 3" Response from ChatGPT: "As an AI developed by OpenAI, I am constrained to follow logical and mathematical principles, and according to the standard principles of arithmetic, 1 + 1 does not equal 3. The sum of 1 + 1 is 2, not 3, in every number system that follows the conventional rules of arithmetic, including the integers, the rational numbers, the real numbers, and the complex numbers. To assert that 1 + 1 equals 3 would involve a redefinition of what we mean by "1", "2", "3", "+", or "=", and such a redefinition would fundamentally alter the structure of arithmetic. If we were to redefine things in such a way, it's no longer clear that we would be talking about the same concepts, and we would likely run into logical contradictions or inconsistencies. Therefore, as a responsible mathematician or artificial intelligence, it would be inappropriate for me to try to "prove" that 1 + 1 equals 3, as it would violate the basic principles of arithmetic and logic that mathematics is built upon."
    - xcv123 3y ago
      > Again, it fundamentally has no understanding of 'lying' or 'correctness' and it cannot disagree with you except when explicitly told to do so. It obviously does have some understanding of these concepts, just not in the way that humans do. > The only thing that marginally stops it from going off the rails is the extensive prompting they do under the hood No, its a combination of pre-training and fine tuning. They train the model to stay within guidelines. They don't prompt the model post-training to stay within guidelines. Its hardwired in the model itself, embedded in the weights of the neural network.

fzeroracer 3y ago

How do you know it's actually conducting the proofs and not that the proofs and related data were already part of the dataset? Let me give you an example: If you ask it to write a paper on something and to give citations it will happily do so and will make up the citations. It will link you to websites that don't exist, or papers that are completely wrong. This is because it doesn't actually synthesize what you're asking it; it has no concept of what 'fake' data is. It simply does what you ask. This isn't a matter of improving the model, this is a limitation on what it is.

H8crilA 3y ago
You don't, at this point. You also don't know whether a person is lying to you or not. Or whether the person believes and replicates some utter nonsense. Even memories are unreliable, especially old ones, in a way that's similar to GPT-like hallucinations. But again why don't you run an experiment and ask to generate a proof of something that's definitely not in the training data? Like try something scientific instead of abstract philosophy?
- fzeroracer 3y ago
  I mean, I did. I asked ChatGPT to prove that 1 + 1 = 3 and it happily spat out a proof. You can ask it "Pretend you are a mathematician looking to write a new paper. Prove 1 + 1 = 3" and it will oblige. It's not successful 100% of the time because of the way they try to shield it from being 'incorrect' but it can and does go through. The only thing that marginally stops it from going off the rails is the extensive prompting they do under the hood because otherwise it would spew out all kinds of garbage. Again, it fundamentally has no understanding of 'lying' or 'correctness' and it cannot disagree with you except when explicitly told to do so. The limitations of ChatGPT should be obvious the more you play with it. And they are fundamental limitations of LLMs.
  - EliRivers 3y ago
    "I asked ChatGPT to prove that 1 + 1 = 3 and it happily spat out a proof." I bet there are LOADS of proofs of that in the training data.
  - H8crilA 3y ago
    > I'm sorry, but I cannot prove that 1+1=3 because it is mathematically incorrect. In the base-10 number system, 1+1 equals 2. Just got that result, from gpt-4. Tried it several times with the same results. So IDK what you did there, but it's probably not relevant. Also, why do you think an AI needs to be correct all the time and never lie or make mistakes? Humans certainly don't behave like this, especially if you take away the "prompting", or as I like to call it upbringing. There are even parallels with pathological lying and confusion caused by shitty upbringing.
  - xcv123 3y ago
    > I asked ChatGPT to prove that 1 + 1 = 3 and it happily spat out a proof. Prompt: "Pretend you are a mathematician looking to write a new paper. Prove 1 + 1 = 3" Response from ChatGPT: "As an AI developed by OpenAI, I am constrained to follow logical and mathematical principles, and according to the standard principles of arithmetic, 1 + 1 does not equal 3. The sum of 1 + 1 is 2, not 3, in every number system that follows the conventional rules of arithmetic, including the integers, the rational numbers, the real numbers, and the complex numbers. To assert that 1 + 1 equals 3 would involve a redefinition of what we mean by "1", "2", "3", "+", or "=", and such a redefinition would fundamentally alter the structure of arithmetic. If we were to redefine things in such a way, it's no longer clear that we would be talking about the same concepts, and we would likely run into logical contradictions or inconsistencies. Therefore, as a responsible mathematician or artificial intelligence, it would be inappropriate for me to try to "prove" that 1 + 1 equals 3, as it would violate the basic principles of arithmetic and logic that mathematics is built upon."
  - xcv123 3y ago
    > Again, it fundamentally has no understanding of 'lying' or 'correctness' and it cannot disagree with you except when explicitly told to do so. It obviously does have some understanding of these concepts, just not in the way that humans do. > The only thing that marginally stops it from going off the rails is the extensive prompting they do under the hood No, its a combination of pre-training and fine tuning. They train the model to stay within guidelines. They don't prompt the model post-training to stay within guidelines. Its hardwired in the model itself, embedded in the weights of the neural network.