It might help quite a bit to watch this overview of how ChatGPT works, and especially how it was trained.[1] Human attention was too expensive for the training

It might help quite a bit to watch this overview of how ChatGPT works, and especially how it was trained.[1] Human attention was too expensive for the training, so they made another AI to simulate the trainers. Unfortunately, the trainer simulator training didn't have an expressive enough set of options for the actual humans, which kept a lot of quite valuable feedback out of the whole system by reducing things to single ranking. If the person didn't know the correct answer, they'd likely just go based on gut. This in turn lead the training simulator to do the same. This lead ChatGPT to just try to bullshit an answer rather than stating it didn't know, and getting downvoted. [1] https://www.youtube.com/watch?v=viJt_DXTfwA https://www.youtube.com/watch?v=viJt_DXTfwA