4 ms·

I had a 30 min argument with o1-pro where it was convinced it had solved the halting problem. Tried to gaslight me into thinking I just didn’t understand the su

I had a 30 min argument with o1-pro where it was convinced it had solved the halting problem. Tried to gaslight me into thinking I just didn’t understand the subtlety of the argument. But it’s susceptible to appeal to authority and when I started quoting snippets of textbooks and mathoverflow it finally relented and claimed there had been a “misunderstanding”. It really does argue like a human though now...

radioactivist 2y ago
I had a similar experience with regular o1 about integral that was divergent. It was adamant that it wasn't and would respond to any attempt at persuasion with variants of "its a standard integral" with a "subtle cancellation". When I asked for any source for this standard integral it produced references to support its argument that existed but didn't actually contain the integral. When I told it the references didn't have the result and backpedalled (gaslighting!) to "I never told you they were in there". When I pointed out that in fact it did it insisted this was just a "misunderstanding". It only relented when I told it Mathematica agreed the integral was divergent. It still insisted it never said that the books it pointed to contained this (false, non-sensical) result. This was new behaviour for me to see in an LLM. Usually the problem is these things would just fold when you pushed back. I don't know which is better, but being this confidently wrong (and "lying" when confronted with it) is troubling.
- Animats 2y ago
  > but being this confidently wrong (and "lying" when confronted with it) is troubling. It works in politics, marketing, and self-promotion. If you use the web as a training set, those categories dominate.
  - layer8 2y ago
    Maybe they also trained the model on Sam Altman. ;)
    - vixen99 2y ago
      Please don't crack ones like that when I'm drinking my coffee.
- justatdotin 2y ago
  I've also had it invent non-existent references. > being this confidently wrong (and "lying" when confronted with it) is troubling. I don't find it troubling. I like being reminded to distrust and confirm everything it offers.
  - radioactivist 2y ago
    The troubling part is that the references themselves existed -- one was an obscure Russian text that is difficult to find (but is exactly where you'd expect to find this kind of result, if it existed).
phillipharris 2y ago
This sounds fun to read, can you share the transcript?

radioactivist 2y ago

I had a similar experience with regular o1 about integral that was divergent. It was adamant that it wasn't and would respond to any attempt at persuasion with variants of "its a standard integral" with a "subtle cancellation". When I asked for any source for this standard integral it produced references to support its argument that existed but didn't actually contain the integral. When I told it the references didn't have the result and backpedalled (gaslighting!) to "I never told you they were in there". When I pointed out that in fact it did it insisted this was just a "misunderstanding". It only relented when I told it Mathematica agreed the integral was divergent. It still insisted it never said that the books it pointed to contained this (false, non-sensical) result. This was new behaviour for me to see in an LLM. Usually the problem is these things would just fold when you pushed back. I don't know which is better, but being this confidently wrong (and "lying" when confronted with it) is troubling.

Animats 2y ago
> but being this confidently wrong (and "lying" when confronted with it) is troubling. It works in politics, marketing, and self-promotion. If you use the web as a training set, those categories dominate.
- layer8 2y ago
  Maybe they also trained the model on Sam Altman. ;)
  - vixen99 2y ago
    Please don't crack ones like that when I'm drinking my coffee.
justatdotin 2y ago
I've also had it invent non-existent references. > being this confidently wrong (and "lying" when confronted with it) is troubling. I don't find it troubling. I like being reminded to distrust and confirm everything it offers.
- radioactivist 2y ago
  The troubling part is that the references themselves existed -- one was an obscure Russian text that is difficult to find (but is exactly where you'd expect to find this kind of result, if it existed).

phillipharris 2y ago

This sounds fun to read, can you share the transcript?