"Confidently wrong" is definitely a standard behavior model for LLMs. I agree with the other reply that you're likely to get better results if it has some kind

"Confidently wrong" is definitely a standard behavior model for LLMs. I agree with the other reply that you're likely to get better results if it has some kind of test case to run that's more authoritative than its own reasoning.