It's precisely because it's so amazing that we need to sell it short. It really is basically a statistical completion engine. Auto-suggest on steroids. By so

It's precisely because it's so amazing that we need to sell it short. It really is basically a statistical completion engine. Auto-suggest on steroids. By some interpretations, language models are without any semantic properties. It does not reason. It does not do symbolic manipulation. There is no chain of thought, or argument, as we understand those things, on the way to producing its output. There are no easily-identified objects in its model that map to tokens or objects related to something in the outside world. It is predicting, with some very fancy statistics, the most probable completion to a prompt, based on the patterns in its training input. This is not a dismissal. I think it makes what it can do all the more impressive, and perhaps unsettling. Until recently, I believed something like this could never be accomplished without great advances in more traditional AI - stuff like fuzzy symbolic reasoning and programmed-in general knowledge databases. That ChatGPT lacks such should be kept in mind. Because people seem to frequently fall for the illusion that it is reasoning, that it is doing symbolic manipulation of tokens, etc. It is not. It can't even remember or learn anything! There is no state held between prompt invocations. The architecture just feeds in the recent backlog when you run the next prompt, to help guide it with context. So it's little surprise that with the right prompts it will "lie" and then deny it. It doesn't have a concept of truth nor any memory.