5 ms·

Boss to coworker: "Please do X" Coworker: "mmm, X is important" Boss: "Yes, and I need you to do it" Coworker: "I understand" Boss: "Understanding isn't eno

Boss to coworker: "Please do X" Coworker: "mmm, X is important" Boss: "Yes, and I need you to do it" Coworker: "I understand" Boss: "Understanding isn't enough, say you'll do it" Coworker: "Ok, ok, I will do X" Boss: "Thank you" (leaves). Coworker: returns to what they were doing, does not do X, never had any intention of doing X. That's ChatGPT in some sense - what it's looking for is the right words to make you stop prompting. That's success. It never had any intention of rethinking, or reunderstanding, but it will find some agreement words and rewritten text which have a high probability of making you stop asking. Like the spaceship with a lever on the control board, you flick the lever, spaceship goes into warp drive - wow, having warp drive on your car would be cool, so you unscrew the lever and screw it onto the dashboard of your car. When you flick it, nothing happens. That's ChatGPT in some sense; a complex disconnected lever - disconnected from actions, embodiment, intention, understanding, awareness. A frontend for them, which looks a bit like they look, but missing the mechanisms behind them which implement them.

vikingerik 4y ago
Your last paragraph sounds a lot like something human children might do, taking actions with a superficial mental model that doesn't match what's happening. So give ChatGPT another fifteen years of learning and let's see if it might stop making such mistakes. I'm betting it will.
- RC_ITR 4y ago
  Understanding language and understanding symbols are two very different things. I don’t disagree that we may be able to build symbolic neural nets 15 years from now, but they will look almost nothing like LLMs.
- batty_alex 4y ago
  Just some food for thought: you’re anthropomorphizing a machine, it doesn’t understand like humans do It’s not going to learn more ‘with time,’ either
  - kilgnad 4y ago
    It does. Did you follow the link in my original post. https://www.engraved.blog/building-a-virtual-machine-inside/ https://www.engraved.blog/building-a-virtual-machine-inside/ Read it to the end. The end is quite mind blowing. I'm not sure how someone can finish this article and not realize that chatGPT actually completely understands what you're telling it.
    - jodrellblank 4y ago
      In some ways it's amazing, in other ways it's like saying that if I gave my mother a cue card saying "NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver" and she read that back to me when I said "nvidia-smi" then she must be running a virtual machine in her head and querying it for a GPU. Or if she read "ping reply 14ms" off a notepad she must be simulating ICMP echo and understand networking. She clearly wouldn't be. ChatGPT has zillions of web pages as cue cards. It will have many examples of ping and reply, of first ten prime numbers in Python, of Docker output. Was it actually building a Docker file inside it in any way? I asked it a similar prompt, I'm amazed at how much it can get right, I've never seen any other program or computer do what it does. At the same time, it's not implementing the things behind the scenes that we're asking it to and more complicated prompts makes that visible - it's not actually a CPython runtime or a Linux shell or curl'ing a web page; this is the disconnected lever fallacy I mentioned earlier. "If I copy the interface to a warp drive, that will give me a warp drive, if I copy the interface to a Linux machine, that's the same as being a Linux machine". It's also akin to Searle's Chinese Room - at some point it may be good enough to be indistinguishable from a Linux machine and that would render this argument irrelevant.
      - scarface74 4y ago
        I’ve given it much more complicated prompts to create a specific Python script with very specific requirements and it did pretty well. I’ve even given it a Python script I wrote that took in a JSON file And asked it to make sone changes to the script. It does far more than just mindlessly return results.
      - 4y ago
- idiotsecant 4y ago
  ChatGPT in it's current incarnation will not improve markedly. It's an architectural limitation at this point and more data might smooth out some rough spots, but it won't solve the behavior that this chain is describing. I have no doubt you're right in general, though. My intuition is that a general-purpose cognitive engine that is capable of fully classifying, understanding, and manipulating the world around it will happen in my lifetime, I'm almost sure of it. I can't wait!
  - kilgnad 4y ago
    >ChatGPT in it's current incarnation will not improve markedly. It's an architectural limitation at this point and more data might smooth out some rough spots, but it won't solve the behavior that this chain is describing. Really? how do you know this? Are you an expert in this area and this is actually something experts are talking about or is this just an educated guess? Serious question.
    - freeone3000 4y ago
      I agree with this -- there's no actual mechanism for logical influence at runtime. Logic itself is poorly modelled by studying corpuses -- text sources don't demonstrate it. In addition, GPT is currently saturated with data; it has effectively all of the data. Adding more neurons or more layers might help by adding more memorized facts and exchanges, but I'm of the opinion that our current approach of function approximators fundamentally cannot reason logically. It'll be a while before we can prove or disprove this (the proof techniques used for these are still nascent; a few dozen neurons have been solved...). But I think we'll need another model working in conjunction using a different approach, or even a scaffolding system, as neither the LLM nor the RL on top can implement or enforce logic.
      - kilgnad 4y ago
        >In addition, GPT is currently saturated with data; it has effectively all of the data. No this is not true. The reason why chatGPT is better then GPT-3 is because it's trained on additional reinforcement data to determine good and bad answers. They basically outsourced and hired a bunch of Kenyans to do this. If they hire say a computer scientist to reinforce the code and rate the best coded answers then it can improve in that specific specialty. There is ALOT of room for additional sets of this type of data. Are you sure you know this as an expert in this area or you're just agreeing with him as a layman? I'm certainly a layman myself.
- Xelynega 4y ago
  Just like if you give your car a few decades it might learn to fly?
kilgnad 4y ago
Did you look at the link in my post? It is literally doing the opposite of what you're saying here. Seriously read it to the end. Each task gets more and more complex and chatGPT successfully executes the task in a way that's very mind blowing at the end. https://www.engraved.blog/building-a-virtual-machine-inside/ https://www.engraved.blog/building-a-virtual-machine-inside/ I think most LLMs do what you say. chatGPT is somewhat of an exception. It sometimes does what you describe here but often it doesn't. I think a lot of people are projecting their idea of what LLMs typically do without realizing that chatGPT is actually different.
- lelandbatey 4y ago
  If I give chatGPT the exact text of that post, but do something like change the input of e.g. the primes generating program, but I then change it slightly so that it doesn't work in the real world, ChatGPT doesn't catch the error and instead returns the data as though I put in a working program.
  - jodrellblank 4y ago
    A hudred and fifty odd years ago, Charles Babbage said about his Difference Engine: "On two occasions I have been asked, 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question." voila, ChatGPT. [1] https://en.wikipedia.org/wiki/Difference_engine https://en.wikipedia.org/wiki/Difference_engine
  - kilgnad 4y ago
    Either chatGPT made a careless error, or it lied to you. Both lying and making careless errors use to be both exclusive abilities of human beings.
    - threeseed 4y ago
      The issue is that ChatGPT doesn't know whether it made a careless error or lied to you. And it has no convictions so you can simply inform it that it was telling the truth.

vikingerik 4y ago

Your last paragraph sounds a lot like something human children might do, taking actions with a superficial mental model that doesn't match what's happening. So give ChatGPT another fifteen years of learning and let's see if it might stop making such mistakes. I'm betting it will.

RC_ITR 4y ago
Understanding language and understanding symbols are two very different things. I don’t disagree that we may be able to build symbolic neural nets 15 years from now, but they will look almost nothing like LLMs.
batty_alex 4y ago
Just some food for thought: you’re anthropomorphizing a machine, it doesn’t understand like humans do It’s not going to learn more ‘with time,’ either
- kilgnad 4y ago
  It does. Did you follow the link in my original post. https://www.engraved.blog/building-a-virtual-machine-inside/ https://www.engraved.blog/building-a-virtual-machine-inside/ Read it to the end. The end is quite mind blowing. I'm not sure how someone can finish this article and not realize that chatGPT actually completely understands what you're telling it.
  - jodrellblank 4y ago
    In some ways it's amazing, in other ways it's like saying that if I gave my mother a cue card saying "NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver" and she read that back to me when I said "nvidia-smi" then she must be running a virtual machine in her head and querying it for a GPU. Or if she read "ping reply 14ms" off a notepad she must be simulating ICMP echo and understand networking. She clearly wouldn't be. ChatGPT has zillions of web pages as cue cards. It will have many examples of ping and reply, of first ten prime numbers in Python, of Docker output. Was it actually building a Docker file inside it in any way? I asked it a similar prompt, I'm amazed at how much it can get right, I've never seen any other program or computer do what it does. At the same time, it's not implementing the things behind the scenes that we're asking it to and more complicated prompts makes that visible - it's not actually a CPython runtime or a Linux shell or curl'ing a web page; this is the disconnected lever fallacy I mentioned earlier. "If I copy the interface to a warp drive, that will give me a warp drive, if I copy the interface to a Linux machine, that's the same as being a Linux machine". It's also akin to Searle's Chinese Room - at some point it may be good enough to be indistinguishable from a Linux machine and that would render this argument irrelevant.
    - scarface74 4y ago
      I’ve given it much more complicated prompts to create a specific Python script with very specific requirements and it did pretty well. I’ve even given it a Python script I wrote that took in a JSON file And asked it to make sone changes to the script. It does far more than just mindlessly return results.
    - 4y ago
idiotsecant 4y ago
ChatGPT in it's current incarnation will not improve markedly. It's an architectural limitation at this point and more data might smooth out some rough spots, but it won't solve the behavior that this chain is describing. I have no doubt you're right in general, though. My intuition is that a general-purpose cognitive engine that is capable of fully classifying, understanding, and manipulating the world around it will happen in my lifetime, I'm almost sure of it. I can't wait!
- kilgnad 4y ago
  >ChatGPT in it's current incarnation will not improve markedly. It's an architectural limitation at this point and more data might smooth out some rough spots, but it won't solve the behavior that this chain is describing. Really? how do you know this? Are you an expert in this area and this is actually something experts are talking about or is this just an educated guess? Serious question.
  - freeone3000 4y ago
    I agree with this -- there's no actual mechanism for logical influence at runtime. Logic itself is poorly modelled by studying corpuses -- text sources don't demonstrate it. In addition, GPT is currently saturated with data; it has effectively all of the data. Adding more neurons or more layers might help by adding more memorized facts and exchanges, but I'm of the opinion that our current approach of function approximators fundamentally cannot reason logically. It'll be a while before we can prove or disprove this (the proof techniques used for these are still nascent; a few dozen neurons have been solved...). But I think we'll need another model working in conjunction using a different approach, or even a scaffolding system, as neither the LLM nor the RL on top can implement or enforce logic.
    - kilgnad 4y ago
      >In addition, GPT is currently saturated with data; it has effectively all of the data. No this is not true. The reason why chatGPT is better then GPT-3 is because it's trained on additional reinforcement data to determine good and bad answers. They basically outsourced and hired a bunch of Kenyans to do this. If they hire say a computer scientist to reinforce the code and rate the best coded answers then it can improve in that specific specialty. There is ALOT of room for additional sets of this type of data. Are you sure you know this as an expert in this area or you're just agreeing with him as a layman? I'm certainly a layman myself.
Xelynega 4y ago
Just like if you give your car a few decades it might learn to fly?

kilgnad 4y ago

Did you look at the link in my post? It is literally doing the opposite of what you're saying here. Seriously read it to the end. Each task gets more and more complex and chatGPT successfully executes the task in a way that's very mind blowing at the end. https://www.engraved.blog/building-a-virtual-machine-inside/ https://www.engraved.blog/building-a-virtual-machine-inside/ I think most LLMs do what you say. chatGPT is somewhat of an exception. It sometimes does what you describe here but often it doesn't. I think a lot of people are projecting their idea of what LLMs typically do without realizing that chatGPT is actually different.

lelandbatey 4y ago
If I give chatGPT the exact text of that post, but do something like change the input of e.g. the primes generating program, but I then change it slightly so that it doesn't work in the real world, ChatGPT doesn't catch the error and instead returns the data as though I put in a working program.
- jodrellblank 4y ago
  A hudred and fifty odd years ago, Charles Babbage said about his Difference Engine: "On two occasions I have been asked, 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question." voila, ChatGPT. [1] https://en.wikipedia.org/wiki/Difference_engine https://en.wikipedia.org/wiki/Difference_engine
- kilgnad 4y ago
  Either chatGPT made a careless error, or it lied to you. Both lying and making careless errors use to be both exclusive abilities of human beings.
  - threeseed 4y ago
    The issue is that ChatGPT doesn't know whether it made a careless error or lied to you. And it has no convictions so you can simply inform it that it was telling the truth.