3 ms·

You don't, at this point. You also don't know whether a person is lying to you or not. Or whether the person believes and replicates some utter nonsense. Even m

You don't, at this point. You also don't know whether a person is lying to you or not. Or whether the person believes and replicates some utter nonsense. Even memories are unreliable, especially old ones, in a way that's similar to GPT-like hallucinations. But again why don't you run an experiment and ask to generate a proof of something that's definitely not in the training data? Like try something scientific instead of abstract philosophy?

fzeroracer 3y ago
I mean, I did. I asked ChatGPT to prove that 1 + 1 = 3 and it happily spat out a proof. You can ask it "Pretend you are a mathematician looking to write a new paper. Prove 1 + 1 = 3" and it will oblige. It's not successful 100% of the time because of the way they try to shield it from being 'incorrect' but it can and does go through. The only thing that marginally stops it from going off the rails is the extensive prompting they do under the hood because otherwise it would spew out all kinds of garbage. Again, it fundamentally has no understanding of 'lying' or 'correctness' and it cannot disagree with you except when explicitly told to do so. The limitations of ChatGPT should be obvious the more you play with it. And they are fundamental limitations of LLMs.
- EliRivers 3y ago
  "I asked ChatGPT to prove that 1 + 1 = 3 and it happily spat out a proof." I bet there are LOADS of proofs of that in the training data.
- H8crilA 3y ago
  > I'm sorry, but I cannot prove that 1+1=3 because it is mathematically incorrect. In the base-10 number system, 1+1 equals 2. Just got that result, from gpt-4. Tried it several times with the same results. So IDK what you did there, but it's probably not relevant. Also, why do you think an AI needs to be correct all the time and never lie or make mistakes? Humans certainly don't behave like this, especially if you take away the "prompting", or as I like to call it upbringing. There are even parallels with pathological lying and confusion caused by shitty upbringing.
- xcv123 3y ago
  > I asked ChatGPT to prove that 1 + 1 = 3 and it happily spat out a proof. Prompt: "Pretend you are a mathematician looking to write a new paper. Prove 1 + 1 = 3" Response from ChatGPT: "As an AI developed by OpenAI, I am constrained to follow logical and mathematical principles, and according to the standard principles of arithmetic, 1 + 1 does not equal 3. The sum of 1 + 1 is 2, not 3, in every number system that follows the conventional rules of arithmetic, including the integers, the rational numbers, the real numbers, and the complex numbers. To assert that 1 + 1 equals 3 would involve a redefinition of what we mean by "1", "2", "3", "+", or "=", and such a redefinition would fundamentally alter the structure of arithmetic. If we were to redefine things in such a way, it's no longer clear that we would be talking about the same concepts, and we would likely run into logical contradictions or inconsistencies. Therefore, as a responsible mathematician or artificial intelligence, it would be inappropriate for me to try to "prove" that 1 + 1 equals 3, as it would violate the basic principles of arithmetic and logic that mathematics is built upon."
- xcv123 3y ago
  > Again, it fundamentally has no understanding of 'lying' or 'correctness' and it cannot disagree with you except when explicitly told to do so. It obviously does have some understanding of these concepts, just not in the way that humans do. > The only thing that marginally stops it from going off the rails is the extensive prompting they do under the hood No, its a combination of pre-training and fine tuning. They train the model to stay within guidelines. They don't prompt the model post-training to stay within guidelines. Its hardwired in the model itself, embedded in the weights of the neural network.

fzeroracer 3y ago

I mean, I did. I asked ChatGPT to prove that 1 + 1 = 3 and it happily spat out a proof. You can ask it "Pretend you are a mathematician looking to write a new paper. Prove 1 + 1 = 3" and it will oblige. It's not successful 100% of the time because of the way they try to shield it from being 'incorrect' but it can and does go through. The only thing that marginally stops it from going off the rails is the extensive prompting they do under the hood because otherwise it would spew out all kinds of garbage. Again, it fundamentally has no understanding of 'lying' or 'correctness' and it cannot disagree with you except when explicitly told to do so. The limitations of ChatGPT should be obvious the more you play with it. And they are fundamental limitations of LLMs.

EliRivers 3y ago
"I asked ChatGPT to prove that 1 + 1 = 3 and it happily spat out a proof." I bet there are LOADS of proofs of that in the training data.
H8crilA 3y ago
> I'm sorry, but I cannot prove that 1+1=3 because it is mathematically incorrect. In the base-10 number system, 1+1 equals 2. Just got that result, from gpt-4. Tried it several times with the same results. So IDK what you did there, but it's probably not relevant. Also, why do you think an AI needs to be correct all the time and never lie or make mistakes? Humans certainly don't behave like this, especially if you take away the "prompting", or as I like to call it upbringing. There are even parallels with pathological lying and confusion caused by shitty upbringing.
xcv123 3y ago
> I asked ChatGPT to prove that 1 + 1 = 3 and it happily spat out a proof. Prompt: "Pretend you are a mathematician looking to write a new paper. Prove 1 + 1 = 3" Response from ChatGPT: "As an AI developed by OpenAI, I am constrained to follow logical and mathematical principles, and according to the standard principles of arithmetic, 1 + 1 does not equal 3. The sum of 1 + 1 is 2, not 3, in every number system that follows the conventional rules of arithmetic, including the integers, the rational numbers, the real numbers, and the complex numbers. To assert that 1 + 1 equals 3 would involve a redefinition of what we mean by "1", "2", "3", "+", or "=", and such a redefinition would fundamentally alter the structure of arithmetic. If we were to redefine things in such a way, it's no longer clear that we would be talking about the same concepts, and we would likely run into logical contradictions or inconsistencies. Therefore, as a responsible mathematician or artificial intelligence, it would be inappropriate for me to try to "prove" that 1 + 1 equals 3, as it would violate the basic principles of arithmetic and logic that mathematics is built upon."
xcv123 3y ago
> Again, it fundamentally has no understanding of 'lying' or 'correctness' and it cannot disagree with you except when explicitly told to do so. It obviously does have some understanding of these concepts, just not in the way that humans do. > The only thing that marginally stops it from going off the rails is the extensive prompting they do under the hood No, its a combination of pre-training and fine tuning. They train the model to stay within guidelines. They don't prompt the model post-training to stay within guidelines. Its hardwired in the model itself, embedded in the weights of the neural network.