>a genuinely scary and unsettling experience for me Every time I think of a good thing to ask ChatGPT (free version), it gives me garbage. Not only for program

>a genuinely scary and unsettling experience for me Every time I think of a good thing to ask ChatGPT (free version), it gives me garbage. Not only for programming tasks, but also for, basically, short essays in English on some topic. By garbage, I mean superficially polished but fundamentally incoherent when you pay attention to the details. It simply doesn't grasp the topic, because, I think, there is no mechanism for that. I don't see a whit of difference between code and plain English - they both require something it doesn't have available. Hypothesis #1 is that I'm just shitty at prompting it. Hypothesis #2, that I like better for reasons of vanity, is that I think of new ideas or questions constantly and most people do not. If you ask anything that exists on the internet then you will be tricked. >when ChatGPT does it I just don't understand how it's even possible for it to make a mistake like that while it clearly is quite on-top of the rest of the problem-space It literally only makes plausible things, true or false. It generates falsehoods that are adjacent to truths or at least to human-produced utterances, in an abstract space. That space naturally has a huge number of falsehoods swirling around every fact it digested. How is that not intuitive, and intuitively so dangerous as to be practically useless? It is inherently the best bullshit, the most superhumanly crafted bullshit, by the much quoted definition of bullshit I first read on HN. But I do keep trying it again and again, at intervals, to see what it has to say.