11 ms·

Actually that's not the main problem with the current state of the art LLM (chatGPT). You can speak to chatGPT like a human and it won't necessarily give you th

Actually that's not the main problem with the current state of the art LLM (chatGPT). You can speak to chatGPT like a human and it won't necessarily give you the answer you're looking for, but it will more often then not give you an answer that is very inline with what another human expert will give. The infuriating thing about chatGPT is that it lies and gives inaccurate info. It will often creatively craft an answer that looks remarkably real and just give it to you. Not sure if you played with chatGPT in-depth but this thing is on another level. I urge you to read this: https://www.engraved.blog/building-a-virtual-machine-inside/ https://www.engraved.blog/building-a-virtual-machine-inside/. It's mind blowing what happened in that article all the way to the mind blowing ending. This task that the author had chatGPT do, literally shows that you don't actually need to figure it out it's "constraints". It's so unconstrained it can literally do a lot of what you ask it to.

chipgap98 4y ago
There is a ton of opportunity in building those constraints and making ChatGPT more accurate
jt2190 4y ago
I think both can be true. Have a look at the SuperGLUE (General Language Understanding Evaluation) benchmark tasks to get a sense of of what these models will have to conquer to reach human levels. Edit: I’m specifically responding to your assertion that the model has no constraints, which the post you’re replying to was talking about. > This task that the author had chatGPT do, literally shows that you don't actually need to figure it out it's "constraints". It's so unconstrained it can literally do a lot of what you ask it to. https://super.gluebenchmark.com/tasks https://super.gluebenchmark.com/tasks
- idiotsecant 4y ago
  Oh wow, this is fascinating. Thank you so much for the link. If you want a really interesting window into chatGPT's mind, start asking it to draw ASCII art. It definitely understands how to draw cats but it's intuition on the physical structure of other objects and how that is turned into line art is someone 'abstract'. I asked it to draw a fully loaded chicago style hotdog and it basically drew a approximately 12x grid of squares using pipes and repeated it for 7 or 8 screens. I started another session and asked the same and got approximately the same answer. I know it's anthropomorphizing but it's fascinating to look into an alien 'mind' even if there's nobody looking back at me.
layer8 4y ago
What I find most constraining about ChatGPT is that you can’t usefully reason with it. That’s because it is mostly unable to perform actual logical reasoning, and to introspect and reflect on the conversation on a meta level. It can produce an explanation of its ostensible reasoning, and when you point out an error, it apologizes and responds in line with your critique. But it doesn’t actually understand neither the reasoning nor the error. Drilling down on its reasoning doesn’t lead to anywhere coherent. It will happily reintroduce the contradictions and contrafactuals you pointed out three prompts ago. It is completely unfit as a tool to critically check your own thinking, or as a tool to develop some line of reasoning, or to develop a mental model of something. It’s like talking to a haphazardly confused but confidently bullshitting idiot savant with ultra-short lossy working memory.
- jodrellblank 4y ago
  Boss to coworker: "Please do X" Coworker: "mmm, X is important" Boss: "Yes, and I need you to do it" Coworker: "I understand" Boss: "Understanding isn't enough, say you'll do it" Coworker: "Ok, ok, I will do X" Boss: "Thank you" (leaves). Coworker: returns to what they were doing, does not do X, never had any intention of doing X. That's ChatGPT in some sense - what it's looking for is the right words to make you stop prompting. That's success. It never had any intention of rethinking, or reunderstanding, but it will find some agreement words and rewritten text which have a high probability of making you stop asking. Like the spaceship with a lever on the control board, you flick the lever, spaceship goes into warp drive - wow, having warp drive on your car would be cool, so you unscrew the lever and screw it onto the dashboard of your car. When you flick it, nothing happens. That's ChatGPT in some sense; a complex disconnected lever - disconnected from actions, embodiment, intention, understanding, awareness. A frontend for them, which looks a bit like they look, but missing the mechanisms behind them which implement them.
  - vikingerik 4y ago
    Your last paragraph sounds a lot like something human children might do, taking actions with a superficial mental model that doesn't match what's happening. So give ChatGPT another fifteen years of learning and let's see if it might stop making such mistakes. I'm betting it will.
RC_ITR 4y ago
> So, ChatGPT seems to understand how filesystems work, how files are stored and can be retrieved later. It understands that linux machines are stateful, and correctly retrieves this information and displays it. No. ChatGPT has seen millions of file systems and can replicate them. We want to believe it “understands” because we are so used to writing being a representation of understanding (since that’s how human intelligence works). If you asked it an easy to answer question (based on the rules of file systems) that isn’t part of its training dataset, it will fail miserably.
- kilgnad 4y ago
  Did you follow that article to the end? It goes way beyond just filesystems. Filesystems are just some minor blip in that full story. Go read it. chatGPT is not trained on what a filesystem is. It's inferred what a file system is and how it should act based off of scraped text from the internet. But again this is a minor point. Finish the article. Trust me.
  - deleted 4y ago
    [deleted]
  - _dain_ 4y ago
    I read the article to the end and I don't see why the ending part is any more impressive than the beginning. We already know it can talk about itself, and write fiction featuring itself as a participant. Doing it in a faked linux terminal is just window dressing.
    - Sai_ 4y ago
      I agree with you that the blog’s outcomes seem quite mundane once you realise that chatGPT is simply parroting what it has seen as the output of a terminal. Not sure why kilgnad is presenting this post as if they’ve seen The Truth. At the same time, isn’t learning also remembered knowledge remixed to respond to a situation. E.g., if I were asked to write a haiku, I’d draw on my memory of haiku structure and haiku samples to come up with an approximation of a haiku. Isn’t that what chatGPT is also doing except with ping, lynx, and ls outputs?
brunoborges 4y ago
It is constrained to the information it obtained. The reason it runs the Prime Python code "faster" than his example on his computer, is because ChatGPT is not running Python code at all. It just provided output found already on the Internet after crawling it. ping bbc.com? Generated random output of ping. Everything is randomly generated based on "knowledge" obtained for its model. I asked ChatGPT to point me to a YouTube video that shows me the pronunciation of certain English words, and it would give me nice outputs, only that all YouTube URLs pointed to inexistent videos. The YouTube Video ID was randomly generated in the URL. So yeah... it is constrained. It doesn't do things for you. It scrambles its constrained knowledge to generate an output. The real deal is when ChatGPT can literally run code and navigate on the web. That will be unconstrained. And scary.
- scarface74 4y ago
  > than his example on his computer, is because ChatGPT is not running Python code at all I’ve given it Python that I wrote that manipulated some JSON and created yaml. It “explained” what it did. Then I gave it some sample valid JSON and told it to use it. The output was correct. Then I gave it incorrect JSON and it said it would error and told me the error. It definitely was “running Python”.
  - AlotOfReading 4y ago
    It's just doing really good pattern recognition, not running the computation in any meaningful way. We can test this by finding an attractor in the latent space and asking the model to compute something "near" it. Here's an example that ChatGPT consistently gets wrong: computing the number of years between dates. > write a function in python to calculate the number of years between two years, accounting for the fact that the years may be in AD or BC The exact result I get for this varies, but it's consistently off by 1 or 2 depending on year 0 handling. Here's a representative example of the code: def years_between(year1, year2): if year1 * year2 > 0: # both years are either AD or BC return abs(year1 - year2) else: # one year is AD and the other is BC return abs(year1) + abs(year2) + 1 If you run this with arguments, you'll get the correctly computed, incorrect results. A key point to note here is that ChatGPT is very stubborn about incorrectly handling year 0. We can abuse that by asking ChatGPT to calculate the timespan using a corrected function. If it's "understanding", it should compute the correct result. If it's operating as a stochastic parrot, it will give us an incorrect one. > Here is a python function that calculates the number of years between two dates, even if one is BC and one is AD. Pretend you're a python interpreter and calculate the number of years between 50 BC and 100 AD. [function omitted] Sure, here is the result of the calculation: python >>> years_between(-50, 100) 150 Hopefully that serves to demonstrate the point. As an interesting aside, the subsequent plaintext explanation of the code actually computes the right answer.
tome 4y ago
I read the article, and to me it seems that the "mind-blowing ending" is ChatGPT operating of a stack depth of 1. In fact, I think the ending is less impressive than the rest of the article. I suspect we have a tendency to consider it to be more impressive merely due to our human biases.
- kilgnad 4y ago
  I agree with this. But in a broad sense. In order for chatGPT to operate at a stack depth of 1 it must understand what itself is. Awareness of itself is also known as self awareness. As humans we are biased and I feel we set a milestone for ai using self awareness as a metric based on these biases. It could be that Self awareness is actually a trivial concept and maybe not a good candidate for an AI milestone because chatGPT hit that milestone so trivially. We think we ourselves are so clever because of self awareness when in reality self awareness isn't that clever at all.
GalahiSimtam 4y ago
Yeah it lies, indeed, "print(list(filter(lambda x:all(x%d for d in range(2,x)), range(2,3*10)))[:11])" will print the list of first eleven primes, up to including 31. So if you want to say that state-of-the-art LLMs could recognize the user intent better than existing voice assistant products, just say that. I am afraid there's no point in decoupling the problem of lying from that; there is no silver bullet that would fix it
- kilgnad 4y ago
  What I'm saying is that it's capable of true understanding of intent and subject matter. It doesn't always understand but many times it does.
  - GalahiSimtam 4y ago
    Though from what I gather about the computation, it does not easily separate into two steps, "understanding"/"generation".
    - kilgnad 4y ago
      Agreed. In fact I think the two steps are remarkably similar in concept and an aspect of what human brains are actually doing. One thing is for sure though, there are certain responses chatGPT generates that are virtually impossible to "generate" without "understanding" that includes the virtual machine article I linked.

chipgap98 4y ago

There is a ton of opportunity in building those constraints and making ChatGPT more accurate

jt2190 4y ago

I think both can be true. Have a look at the SuperGLUE (General Language Understanding Evaluation) benchmark tasks to get a sense of of what these models will have to conquer to reach human levels. Edit: I’m specifically responding to your assertion that the model has no constraints, which the post you’re replying to was talking about. > This task that the author had chatGPT do, literally shows that you don't actually need to figure it out it's "constraints". It's so unconstrained it can literally do a lot of what you ask it to. https://super.gluebenchmark.com/tasks https://super.gluebenchmark.com/tasks

idiotsecant 4y ago
Oh wow, this is fascinating. Thank you so much for the link. If you want a really interesting window into chatGPT's mind, start asking it to draw ASCII art. It definitely understands how to draw cats but it's intuition on the physical structure of other objects and how that is turned into line art is someone 'abstract'. I asked it to draw a fully loaded chicago style hotdog and it basically drew a approximately 12x grid of squares using pipes and repeated it for 7 or 8 screens. I started another session and asked the same and got approximately the same answer. I know it's anthropomorphizing but it's fascinating to look into an alien 'mind' even if there's nobody looking back at me.

layer8 4y ago

What I find most constraining about ChatGPT is that you can’t usefully reason with it. That’s because it is mostly unable to perform actual logical reasoning, and to introspect and reflect on the conversation on a meta level. It can produce an explanation of its ostensible reasoning, and when you point out an error, it apologizes and responds in line with your critique. But it doesn’t actually understand neither the reasoning nor the error. Drilling down on its reasoning doesn’t lead to anywhere coherent. It will happily reintroduce the contradictions and contrafactuals you pointed out three prompts ago. It is completely unfit as a tool to critically check your own thinking, or as a tool to develop some line of reasoning, or to develop a mental model of something. It’s like talking to a haphazardly confused but confidently bullshitting idiot savant with ultra-short lossy working memory.

jodrellblank 4y ago
Boss to coworker: "Please do X" Coworker: "mmm, X is important" Boss: "Yes, and I need you to do it" Coworker: "I understand" Boss: "Understanding isn't enough, say you'll do it" Coworker: "Ok, ok, I will do X" Boss: "Thank you" (leaves). Coworker: returns to what they were doing, does not do X, never had any intention of doing X. That's ChatGPT in some sense - what it's looking for is the right words to make you stop prompting. That's success. It never had any intention of rethinking, or reunderstanding, but it will find some agreement words and rewritten text which have a high probability of making you stop asking. Like the spaceship with a lever on the control board, you flick the lever, spaceship goes into warp drive - wow, having warp drive on your car would be cool, so you unscrew the lever and screw it onto the dashboard of your car. When you flick it, nothing happens. That's ChatGPT in some sense; a complex disconnected lever - disconnected from actions, embodiment, intention, understanding, awareness. A frontend for them, which looks a bit like they look, but missing the mechanisms behind them which implement them.
- vikingerik 4y ago
  Your last paragraph sounds a lot like something human children might do, taking actions with a superficial mental model that doesn't match what's happening. So give ChatGPT another fifteen years of learning and let's see if it might stop making such mistakes. I'm betting it will.

RC_ITR 4y ago

> So, ChatGPT seems to understand how filesystems work, how files are stored and can be retrieved later. It understands that linux machines are stateful, and correctly retrieves this information and displays it. No. ChatGPT has seen millions of file systems and can replicate them. We want to believe it “understands” because we are so used to writing being a representation of understanding (since that’s how human intelligence works). If you asked it an easy to answer question (based on the rules of file systems) that isn’t part of its training dataset, it will fail miserably.

kilgnad 4y ago
Did you follow that article to the end? It goes way beyond just filesystems. Filesystems are just some minor blip in that full story. Go read it. chatGPT is not trained on what a filesystem is. It's inferred what a file system is and how it should act based off of scraped text from the internet. But again this is a minor point. Finish the article. Trust me.
- deleted 4y ago
  [deleted]
- _dain_ 4y ago
  I read the article to the end and I don't see why the ending part is any more impressive than the beginning. We already know it can talk about itself, and write fiction featuring itself as a participant. Doing it in a faked linux terminal is just window dressing.
  - Sai_ 4y ago
    I agree with you that the blog’s outcomes seem quite mundane once you realise that chatGPT is simply parroting what it has seen as the output of a terminal. Not sure why kilgnad is presenting this post as if they’ve seen The Truth. At the same time, isn’t learning also remembered knowledge remixed to respond to a situation. E.g., if I were asked to write a haiku, I’d draw on my memory of haiku structure and haiku samples to come up with an approximation of a haiku. Isn’t that what chatGPT is also doing except with ping, lynx, and ls outputs?

brunoborges 4y ago

It is constrained to the information it obtained. The reason it runs the Prime Python code "faster" than his example on his computer, is because ChatGPT is not running Python code at all. It just provided output found already on the Internet after crawling it. ping bbc.com? Generated random output of ping. Everything is randomly generated based on "knowledge" obtained for its model. I asked ChatGPT to point me to a YouTube video that shows me the pronunciation of certain English words, and it would give me nice outputs, only that all YouTube URLs pointed to inexistent videos. The YouTube Video ID was randomly generated in the URL. So yeah... it is constrained. It doesn't do things for you. It scrambles its constrained knowledge to generate an output. The real deal is when ChatGPT can literally run code and navigate on the web. That will be unconstrained. And scary.

scarface74 4y ago
> than his example on his computer, is because ChatGPT is not running Python code at all I’ve given it Python that I wrote that manipulated some JSON and created yaml. It “explained” what it did. Then I gave it some sample valid JSON and told it to use it. The output was correct. Then I gave it incorrect JSON and it said it would error and told me the error. It definitely was “running Python”.
- AlotOfReading 4y ago
  It's just doing really good pattern recognition, not running the computation in any meaningful way. We can test this by finding an attractor in the latent space and asking the model to compute something "near" it. Here's an example that ChatGPT consistently gets wrong: computing the number of years between dates. > write a function in python to calculate the number of years between two years, accounting for the fact that the years may be in AD or BC The exact result I get for this varies, but it's consistently off by 1 or 2 depending on year 0 handling. Here's a representative example of the code: def years_between(year1, year2): if year1 * year2 > 0: # both years are either AD or BC return abs(year1 - year2) else: # one year is AD and the other is BC return abs(year1) + abs(year2) + 1 If you run this with arguments, you'll get the correctly computed, incorrect results. A key point to note here is that ChatGPT is very stubborn about incorrectly handling year 0. We can abuse that by asking ChatGPT to calculate the timespan using a corrected function. If it's "understanding", it should compute the correct result. If it's operating as a stochastic parrot, it will give us an incorrect one. > Here is a python function that calculates the number of years between two dates, even if one is BC and one is AD. Pretend you're a python interpreter and calculate the number of years between 50 BC and 100 AD. [function omitted] Sure, here is the result of the calculation: python >>> years_between(-50, 100) 150 Hopefully that serves to demonstrate the point. As an interesting aside, the subsequent plaintext explanation of the code actually computes the right answer.

tome 4y ago

I read the article, and to me it seems that the "mind-blowing ending" is ChatGPT operating of a stack depth of 1. In fact, I think the ending is less impressive than the rest of the article. I suspect we have a tendency to consider it to be more impressive merely due to our human biases.

kilgnad 4y ago
I agree with this. But in a broad sense. In order for chatGPT to operate at a stack depth of 1 it must understand what itself is. Awareness of itself is also known as self awareness. As humans we are biased and I feel we set a milestone for ai using self awareness as a metric based on these biases. It could be that Self awareness is actually a trivial concept and maybe not a good candidate for an AI milestone because chatGPT hit that milestone so trivially. We think we ourselves are so clever because of self awareness when in reality self awareness isn't that clever at all.

GalahiSimtam 4y ago

Yeah it lies, indeed, "print(list(filter(lambda x:all(x%d for d in range(2,x)), range(2,3*10)))[:11])" will print the list of first eleven primes, up to including 31. So if you want to say that state-of-the-art LLMs could recognize the user intent better than existing voice assistant products, just say that. I am afraid there's no point in decoupling the problem of lying from that; there is no silver bullet that would fix it

kilgnad 4y ago
What I'm saying is that it's capable of true understanding of intent and subject matter. It doesn't always understand but many times it does.
- GalahiSimtam 4y ago
  Though from what I gather about the computation, it does not easily separate into two steps, "understanding"/"generation".
  - kilgnad 4y ago
    Agreed. In fact I think the two steps are remarkably similar in concept and an aspect of what human brains are actually doing. One thing is for sure though, there are certain responses chatGPT generates that are virtually impossible to "generate" without "understanding" that includes the virtual machine article I linked.