At first I was curious if OP tried arguing with it or telling it the correct answer. If he just asked the question without any further interaction, it would sti

At first I was curious if OP tried arguing with it or telling it the correct answer. If he just asked the question without any further interaction, it would still have no data that it generated the wrong answer. But as another comment says, many people have tried similar things and posted about it online, that information is siphoned by now and the LLMs weights get updated.