7 ms·
What I find most constraining about ChatGPT is that you can’t usefully reason with it. That’s because it is mostly unable to perform actual logical reasoning, a
by layer8 4y ago
What I find most constraining about ChatGPT is that you can’t usefully reason with it. That’s because it is mostly unable to perform actual logical reasoning, and to introspect and reflect on the conversation on a meta level. It can produce an explanation of its ostensible reasoning, and when you point out an error, it apologizes and responds in line with your critique. But it doesn’t actually understand neither the reasoning nor the error. Drilling down on its reasoning doesn’t lead to anywhere coherent. It will happily reintroduce the contradictions and contrafactuals you pointed out three prompts ago. It is completely unfit as a tool to critically check your own thinking, or as a tool to develop some line of reasoning, or to develop a mental model of something.
It’s like talking to a haphazardly confused but confidently bullshitting idiot savant with ultra-short lossy working memory.
- jodrellblank 4y agoBoss to coworker: "Please do X" Coworker: "mmm, X is important" Boss: "Yes, and I need you to do it" Coworker: "I understand" Boss: "Understanding isn't enough, say you'll do it" Coworker: "Ok, ok, I will do X" Boss: "Thank you" (leaves). Coworker: returns to what they were doing, does not do X, never had any intention of doing X. That's ChatGPT in some sense - what it's looking for is the right words to make you stop prompting. That's success. It never had any intention of rethinking, or reunderstanding, but it will find some agreement words and rewritten text which have a high probability of making you stop asking. Like the spaceship with a lever on the control board, you flick the lever, spaceship goes into warp drive - wow, having warp drive on your car would be cool, so you unscrew the lever and screw it onto the dashboard of your car. When you flick it, nothing happens. That's ChatGPT in some sense; a complex disconnected lever - disconnected from actions, embodiment, intention, understanding, awareness. A frontend for them, which looks a bit like they look, but missing the mechanisms behind them which implement them.
- vikingerik 4y agoYour last paragraph sounds a lot like something human children might do, taking actions with a superficial mental model that doesn't match what's happening. So give ChatGPT another fifteen years of learning and let's see if it might stop making such mistakes. I'm betting it will.
- RC_ITR 4y agoUnderstanding language and understanding symbols are two very different things. I don’t disagree that we may be able to build symbolic neural nets 15 years from now, but they will look almost nothing like LLMs.
- batty_alex 4y agoJust some food for thought: you’re anthropomorphizing a machine, it doesn’t understand like humans do It’s not going to learn more ‘with time,’ either
- kilgnad 4y agoIt does. Did you follow the link in my original post. https://www.engraved.blog/building-a-virtual-machine-inside/ https://www.engraved.blog/building-a-virtual-machine-inside/ Read it to the end. The end is quite mind blowing. I'm not sure how someone can finish this article and not realize that chatGPT actually completely understands what you're telling it.
- jodrellblank 4y agoIn some ways it's amazing, in other ways it's like saying that if I gave my mother a cue card saying "NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver" and she read that back to me when I said "nvidia-smi" then she must be running a virtual machine in her head and querying it for a GPU. Or if she read "ping reply 14ms" off a notepad she must be simulating ICMP echo and understand networking. She clearly wouldn't be. ChatGPT has zillions of web pages as cue cards. It will have many examples of ping and reply, of first ten prime numbers in Python, of Docker output. Was it actually building a Docker file inside it in any way? I asked it a similar prompt, I'm amazed at how much it can get right, I've never seen any other program or computer do what it does. At the same time, it's not implementing the things behind the scenes that we're asking it to and more complicated prompts makes that visible - it's not actually a CPython runtime or a Linux shell or curl'ing a web page; this is the disconnected lever fallacy I mentioned earlier. "If I copy the interface to a warp drive, that will give me a warp drive, if I copy the interface to a Linux machine, that's the same as being a Linux machine". It's also akin to Searle's Chinese Room - at some point it may be good enough to be indistinguishable from a Linux machine and that would render this argument irrelevant.
- kilgnad 4y agoDid you look at the link in my post? It is literally doing the opposite of what you're saying here. Seriously read it to the end. Each task gets more and more complex and chatGPT successfully executes the task in a way that's very mind blowing at the end. https://www.engraved.blog/building-a-virtual-machine-inside/ https://www.engraved.blog/building-a-virtual-machine-inside/ I think most LLMs do what you say. chatGPT is somewhat of an exception. It sometimes does what you describe here but often it doesn't. I think a lot of people are projecting their idea of what LLMs typically do without realizing that chatGPT is actually different.
- lelandbatey 4y agoIf I give chatGPT the exact text of that post, but do something like change the input of e.g. the primes generating program, but I then change it slightly so that it doesn't work in the real world, ChatGPT doesn't catch the error and instead returns the data as though I put in a working program.
- jodrellblank 4y agoA hudred and fifty odd years ago, Charles Babbage said about his Difference Engine: "On two occasions I have been asked, 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question." voila, ChatGPT. [1] https://en.wikipedia.org/wiki/Difference_engine https://en.wikipedia.org/wiki/Difference_engine
- kilgnad 4y agoEither chatGPT made a careless error, or it lied to you. Both lying and making careless errors use to be both exclusive abilities of human beings.
- threeseed 4y agoThe issue is that ChatGPT doesn't know whether it made a careless error or lied to you. And it has no convictions so you can simply inform it that it was telling the truth.
- c3534l 4y ago> it apologizes and responds in line with your critique I find it will often double down, requiring me to look it up. Then when I present that, it will find some little corner case where it could be true, prompting me to look that up, too. And then it wild gaslight me, pretending it meant something else, didn't understand the question, or refuse to acknowledge it said what it said. Its an insidious and often subtle liar. There are GOFAI ontology models that I think would actually integrate well into ChatGPT. It's basically solved the language part, but not the AI part, and so it really is more of an interface. So I guess like the OP is talking about. It just needs intelligent systems underneath to interface with.
- idiotsecant 4y agoI've seen that recently too! In fact, it seems like a different behavior than when it first went online. I spend what is probably an unhealthy amount of time using chatGPT because it fascinates me, and I've noticed several instances when it doubled down and flat out insisted that I was wrong when I brought up objections to something that it got wrong. For example, I asked it to produce a python script to give a certain output and the result was a python script that simply would not have produced the output that it described. I executed the program, provided chatGPT with the output, explained the problem, and asked it if it agreed. It insisted that I must have something configured incorrectly in my python environment!! It definitely has a little more sassy in it than it used to.
- Semaphor 4y agoI recently had the new (to me) experience of chatgpt instead deciding that I was wrong instead of it.
- hooande 4y agoIn my experience ChatGPT does reasoning extraordinarily well. The biggest key is that it can explain its thinking clearly. I may or may not agree what it's saying, or it could be outright wrong. But as long as I can see the "step by step" explanation of the thought process, I can make my own informed judgements. And there IS a thought process, that's clear and visible. The scope of topics and concepts it can work with is infinite. Of course there are a large number of edge cases and they are easy to find. But for the 80% of standard conversations, it seems to do just fine. I can have it build a list of pros and cons, rank options, make estimates. About any topic I can imagine, at any time. And all of that is backed by all of the data on the internet. Personally, I don't spend a lot of time asking it trick questions or philosophical gotchas. When I drill down on its reasoning, I find it to be very coherent. I've had a completely different experience than you.
- kilgnad 4y agoPeople are just latching on to the gotchas and the exceptions. Or they aren't really poking at this thing in-depth. ChatGPT is a different iteration of a language model. It does reasoning and it understands things.
- anonymouskimmer 4y agoOften when asked for their reasoning people will post-hoc a rationale. This post-hoc rationale is usually not the actual reasoning used to originally arrive at a point-of-view though. In one study I recall reading about but can't find at the moment people filled out multiple-choice surveys, then were asked to justify certain choices that the researcher told them they had chosen. They justified these choices even though they had originally chosen a different answer. Also: https://skepticink.com/tippling/2013/11/14/post-hoc-rationalisation-reasoning-our-intuition-and-changing-our-minds/ https://skepticink.com/tippling/2013/11/14/post-hoc-rational... I presume to some extent this is also the case with ChatGPT. I presume this because I don't think most people would understand (or have the time to fully read) the actual process the LLM is using to arrive at a particular output from a particular input. The end result looks kind of human. But is this the post-hoc human, the bullshitter human, or is this something else entirely (i.e. pattern matching - effectively mimicry, not of the human providing the input, but of the corpus)?