6 ms·

It's just a language model. I've asked it to prove the Riemann hypothesis, GPT-4 replied it can't, since none of the training data say anything about such proo

by fer 3y ago

It's just a language model. I've asked it to prove the Riemann hypothesis, GPT-4 replied it can't, since none of the training data say anything about such proof. Then I asked it to prove statements of the implications of said hypothesis (I tried 2 or 3, don't remember which), without calling them by name or mentioning Riemann, and it confidently proved them in a way that it reads like it knows what its doing, though it really doesn't. It just takes bits and pieces from proofs in the same field of mathematics that kinda hold together language-wise.

Buttons840 3y ago
Yeah, I've asked it to evaluate Python loops that are about as complex as FizzBuzz, it makes mistakes and even when I direct its attention towards those mistakes it wasn't able to correct itself.
- exitb 3y ago
  I don’t think it’s a reasonable expectation for a language model to evaluate arbitrary code.
  - daveguy 3y ago
    It's definitely not a reasonable expectation. That doesn't stop the breathless hype from implying it is reasonable.
    - williamcotton 3y ago
      Where is this breathless hype everyone speaks of? All I see is an endless stream of populist hot takes on the exact opposite!
      - Culonavirus 3y ago
        "populist hot takes" haha, I'm sorry, but seeing this kind of language in a "gpt can/can't code" debate on a "hacker" forum is just funny to me
      - goatlover 3y ago
        There was a post on reddit about how all software was going to be replaced by LLMs, there's a podcast talking about how ChatGPT will become the universal interface that can reconfigure itself as needed depending on the API it's using and what the user needs. There are people claiming programmers will be replaced by prompt engineers. The hype has been everywhere the last few months.
    - TeMPOraL 3y ago
      It's not breathless hype, as much as surprise in the fact that it actually does work, way better than it should.
  - waboremo 3y ago
    It isn't, yet it's used that way.
    - krapp 3y ago
      Everyone is worried about doomsday scenarios like runaway AGI and Skynet, meanwhile the real danger of AI turns out to be people naively assuming it works like the computer in Star Trek, that because it can hold a conversation, is has some form of intelligence and awareness.
      - JasonFruit 3y ago
        Please keep saying this over and over. Even people who get that it doesn't work that way don't seem to grasp the danger of others relying on these tools in inappropriate ways.
        krapp 3y ago
        The problem is LLMs are kind of an outside context problem for our culture. We're still arguing over the correct language to even describe what it does ("hallucinate"? "confabulate"? "lie?") because every word we have presupposes the existence of a stateful mind with intent. It's a philosophical zombie, which is so counterintuitive most people can't even accept it as a sensible concept. Between the hype and decades of indoctrination through science fiction about how AI is "supposed" to work, I guess it isn't surprising that this is how things are shaking out. People will learn, the way they learned with the internet, but it will take a few years I think.
      - goatlover 3y ago
        The computer in Star Trek seems to only work like a sentient intelligence when you prompt to the Holodeck a certain way, like make a worthy opponent for Data.
  - Buttons840 3y ago
    It's weird that reasoning through complex mathematical proofs is easier (for both me and GPT) than mentally evaluating a short snippet of code. I've done a BSCS, I've done those tests that ask you what the output of a poorly written and tricky loop is, that stuff hurts my brain. In comparison, a mathematical proof, once you're ready, feels good.
    - Filligree 3y ago
      "Mentally evaluating a short snippet of code" requires remembering the state of that code, then repeatedly overwriting your memory as you step through it. GPT-4 isn't able to do that; it is in a sense purely functional. I've had success getting it to evaluate FizzBuzz, but to do so I told it to write out the state of the program one timestep at a time. https://chat.openai.com/share/c109e946-fb6d-494e-8182-fc93d20bdc7e https://chat.openai.com/share/c109e946-fb6d-494e-8182-fc93d2... ...this is actually 3.5. 4 wouldn't need as much explanation.
      - Verdex 3y ago
        This feels like an interesting insight into not just why LLMs might have trouble understanding some code, but also why people might have trouble understanding some code. I wonder how gpt handles code where multiple objects of the same class call the same methods and each individual object calls the same methods with different threads as well as method local variables, class member variables, and static member variables are all in play. [Being fair to our LLM brethren; us meat sacks are sure to hate that hypothetical code as well.]
        Filligree 3y ago
        If you give me an example, I'll try it. From that description I'm not sure of much more than I hate it. :)
        Verdex 3y ago
        Off the top of my head, something like this comes to mind: class X { private static int _staticCount = 0; private int _memberCount = 0; public uint Stuff(int input) { if (input <= 0) { return (uint)(_staticCount + 1); } var y = new Thread(() => { Interlocked.Add(ref X._staticCount, (int)Stuff(input - 1)); Interlocked.Add(ref _memberCount, (int)Stuff(input - 2)); }); y.Start(); y.Join(); return (uint)(Stuff(input - 1) + _memberCount); } } class Program { public static void Main() { var x = new X(); var y = new X(); var z = new X(); uint resultX = x.Stuff(4); uint resultY = y.Stuff(3); uint resultZ = z.Stuff(2); Console.WriteLine(resultX + resultY + resultZ); } } EDIT: Probably should have setup another way to wait for the threads to end, that way all three X classes could be running at the same time. But perhaps this is a good enough start point. I get 3274511360 when I run it. Although, I had to upgrade everything to uint because I was getting some overflow ... so there might be some of that in the output.
Verdex 3y ago
My takeaway from all of the GPT successes and failure (the linear algebra vs riemann hypothesis cases being one example) is that language and jargon can have built in structural constraints such that by having a good LLM you can get good results because the grammatical structure happens to match reality's structure. But reality is under no constraint to conform to the grammatical structures that we use in order to communicate about related problems or solutions. And I suppose this shouldn't be too surprising considering even logic itself runs into godelian issues. For any consistent logic there exists some statement which is true, but which does not have a proof. Which feels vaguely similar to "for any functional LLM there exists some truth which defies the next statistically likely sequence of tokens".
ftxbro 3y ago
> "It's just a language model." This is true. The base pretrained model is even technically a Markov chain. As they have been getting more and more powerful with more parameters and compute and dataset sizes they've been unlocking more and more powerful cognitive capabilities. > "I've asked it to prove the Riemann hypothesis, GPT-4 replied it can't, since none of the training data say anything about such proof." This is some alignment/docility/lobotomy that the RLHF did. The raw base model would never say it. This is part of their No Tay No Syndey push. > "Then I asked it to prove statements of the implications of said hypothesis (I tried 2 or 3, don't remember which), without calling them by name or mentioning Riemann, and it confidently proved them in a way that it reads like it knows what its doing, though it really doesn't. It just takes bits and pieces from proofs in the same field of mathematics that kinda hold together language-wise." Yes it's true. Even though these LLMs have unlocked so many cognitive capabilities, they are still not yet powerful enough to prove the Riemann hypothesis. They aren't quite yet to the level of scientists who can advance the state of the art in their field. Some will say it's an inherent limitation, but I disagree, for complicated reasons that are easy to make superficial and pithy arguments against.
- mikojan 3y ago
  LLMs do not have cognitive capabilities because they lack consciousness and self-awareness. Cognitive abilities require the ability to perceive and understand the world around us through our senses, emotions, and experiences. These processes are rooted in biological structures such as neurons, synapses, and brain chemistry that enable humans to learn from experience. While LLMs can simulate human-like behavior and decision-making based on algorithms trained on large data sets, they lack awareness or understanding of their actions. They rely on pre-programmed rules or statistical correlations rather than intuition or insight derived from personal experience. Therefore, even though LLMs may seem intelligent in certain contexts like playing games or recognizing objects in images with high accuracy rates- they cannot replicate human cognition since there is no consciousness behind those decisions made by the systems.
  - ftxbro 3y ago
    OK but that's why I wrote 'cognitive capabilities' instead of 'intelligence'. What I meant by 'cognitive capabilities' are all of the objective abilities it has unlocked, like scoring well on standardized tests and psychological tests of theory of mind.
visarga 3y ago
> It just takes bits and pieces from proofs in the same field of mathematics that kinda hold together language-wise. That's how AI research works, you do that first, then run the tests and see the score. If it works, then you publish. There is no real understanding anywhere in this process.

by fer 3y ago

Buttons840 3y ago

Yeah, I've asked it to evaluate Python loops that are about as complex as FizzBuzz, it makes mistakes and even when I direct its attention towards those mistakes it wasn't able to correct itself.

exitb 3y ago
I don’t think it’s a reasonable expectation for a language model to evaluate arbitrary code.
- daveguy 3y ago
  It's definitely not a reasonable expectation. That doesn't stop the breathless hype from implying it is reasonable.
  - williamcotton 3y ago
    Where is this breathless hype everyone speaks of? All I see is an endless stream of populist hot takes on the exact opposite!
    - Culonavirus 3y ago
      "populist hot takes" haha, I'm sorry, but seeing this kind of language in a "gpt can/can't code" debate on a "hacker" forum is just funny to me
    - goatlover 3y ago
      There was a post on reddit about how all software was going to be replaced by LLMs, there's a podcast talking about how ChatGPT will become the universal interface that can reconfigure itself as needed depending on the API it's using and what the user needs. There are people claiming programmers will be replaced by prompt engineers. The hype has been everywhere the last few months.
  - TeMPOraL 3y ago
    It's not breathless hype, as much as surprise in the fact that it actually does work, way better than it should.
- waboremo 3y ago
  It isn't, yet it's used that way.
  - krapp 3y ago
    Everyone is worried about doomsday scenarios like runaway AGI and Skynet, meanwhile the real danger of AI turns out to be people naively assuming it works like the computer in Star Trek, that because it can hold a conversation, is has some form of intelligence and awareness.
    - JasonFruit 3y ago
      Please keep saying this over and over. Even people who get that it doesn't work that way don't seem to grasp the danger of others relying on these tools in inappropriate ways.
      - krapp 3y ago
        The problem is LLMs are kind of an outside context problem for our culture. We're still arguing over the correct language to even describe what it does ("hallucinate"? "confabulate"? "lie?") because every word we have presupposes the existence of a stateful mind with intent. It's a philosophical zombie, which is so counterintuitive most people can't even accept it as a sensible concept. Between the hype and decades of indoctrination through science fiction about how AI is "supposed" to work, I guess it isn't surprising that this is how things are shaking out. People will learn, the way they learned with the internet, but it will take a few years I think.
    - goatlover 3y ago
      The computer in Star Trek seems to only work like a sentient intelligence when you prompt to the Holodeck a certain way, like make a worthy opponent for Data.
- Buttons840 3y ago
  It's weird that reasoning through complex mathematical proofs is easier (for both me and GPT) than mentally evaluating a short snippet of code. I've done a BSCS, I've done those tests that ask you what the output of a poorly written and tricky loop is, that stuff hurts my brain. In comparison, a mathematical proof, once you're ready, feels good.
  - Filligree 3y ago
    "Mentally evaluating a short snippet of code" requires remembering the state of that code, then repeatedly overwriting your memory as you step through it. GPT-4 isn't able to do that; it is in a sense purely functional. I've had success getting it to evaluate FizzBuzz, but to do so I told it to write out the state of the program one timestep at a time. https://chat.openai.com/share/c109e946-fb6d-494e-8182-fc93d20bdc7e https://chat.openai.com/share/c109e946-fb6d-494e-8182-fc93d2... ...this is actually 3.5. 4 wouldn't need as much explanation.
    - Verdex 3y ago
      This feels like an interesting insight into not just why LLMs might have trouble understanding some code, but also why people might have trouble understanding some code. I wonder how gpt handles code where multiple objects of the same class call the same methods and each individual object calls the same methods with different threads as well as method local variables, class member variables, and static member variables are all in play. [Being fair to our LLM brethren; us meat sacks are sure to hate that hypothetical code as well.]
      - Filligree 3y ago
        If you give me an example, I'll try it. From that description I'm not sure of much more than I hate it. :)
        Verdex 3y ago
        Off the top of my head, something like this comes to mind: class X { private static int _staticCount = 0; private int _memberCount = 0; public uint Stuff(int input) { if (input <= 0) { return (uint)(_staticCount + 1); } var y = new Thread(() => { Interlocked.Add(ref X._staticCount, (int)Stuff(input - 1)); Interlocked.Add(ref _memberCount, (int)Stuff(input - 2)); }); y.Start(); y.Join(); return (uint)(Stuff(input - 1) + _memberCount); } } class Program { public static void Main() { var x = new X(); var y = new X(); var z = new X(); uint resultX = x.Stuff(4); uint resultY = y.Stuff(3); uint resultZ = z.Stuff(2); Console.WriteLine(resultX + resultY + resultZ); } } EDIT: Probably should have setup another way to wait for the threads to end, that way all three X classes could be running at the same time. But perhaps this is a good enough start point. I get 3274511360 when I run it. Although, I had to upgrade everything to uint because I was getting some overflow ... so there might be some of that in the output.

Verdex 3y ago

My takeaway from all of the GPT successes and failure (the linear algebra vs riemann hypothesis cases being one example) is that language and jargon can have built in structural constraints such that by having a good LLM you can get good results because the grammatical structure happens to match reality's structure. But reality is under no constraint to conform to the grammatical structures that we use in order to communicate about related problems or solutions. And I suppose this shouldn't be too surprising considering even logic itself runs into godelian issues. For any consistent logic there exists some statement which is true, but which does not have a proof. Which feels vaguely similar to "for any functional LLM there exists some truth which defies the next statistically likely sequence of tokens".

ftxbro 3y ago

> "It's just a language model." This is true. The base pretrained model is even technically a Markov chain. As they have been getting more and more powerful with more parameters and compute and dataset sizes they've been unlocking more and more powerful cognitive capabilities. > "I've asked it to prove the Riemann hypothesis, GPT-4 replied it can't, since none of the training data say anything about such proof." This is some alignment/docility/lobotomy that the RLHF did. The raw base model would never say it. This is part of their No Tay No Syndey push. > "Then I asked it to prove statements of the implications of said hypothesis (I tried 2 or 3, don't remember which), without calling them by name or mentioning Riemann, and it confidently proved them in a way that it reads like it knows what its doing, though it really doesn't. It just takes bits and pieces from proofs in the same field of mathematics that kinda hold together language-wise." Yes it's true. Even though these LLMs have unlocked so many cognitive capabilities, they are still not yet powerful enough to prove the Riemann hypothesis. They aren't quite yet to the level of scientists who can advance the state of the art in their field. Some will say it's an inherent limitation, but I disagree, for complicated reasons that are easy to make superficial and pithy arguments against.

mikojan 3y ago
LLMs do not have cognitive capabilities because they lack consciousness and self-awareness. Cognitive abilities require the ability to perceive and understand the world around us through our senses, emotions, and experiences. These processes are rooted in biological structures such as neurons, synapses, and brain chemistry that enable humans to learn from experience. While LLMs can simulate human-like behavior and decision-making based on algorithms trained on large data sets, they lack awareness or understanding of their actions. They rely on pre-programmed rules or statistical correlations rather than intuition or insight derived from personal experience. Therefore, even though LLMs may seem intelligent in certain contexts like playing games or recognizing objects in images with high accuracy rates- they cannot replicate human cognition since there is no consciousness behind those decisions made by the systems.
- ftxbro 3y ago
  OK but that's why I wrote 'cognitive capabilities' instead of 'intelligence'. What I meant by 'cognitive capabilities' are all of the objective abilities it has unlocked, like scoring well on standardized tests and psychological tests of theory of mind.

visarga 3y ago

> It just takes bits and pieces from proofs in the same field of mathematics that kinda hold together language-wise. That's how AI research works, you do that first, then run the tests and see the score. If it works, then you publish. There is no real understanding anywhere in this process.