5 ms·

The author seems to forget that while an AI can create text that looks like code, it can never prove the code is working correctly (see halting problem). As wi

by beders 4y ago

The author seems to forget that while an AI can create text that looks like code, it can never prove the code is working correctly (see halting problem). As with many things, AI will excel at the simple tasks that just require pattern matching and generation to work. Nothing more but also nothing less. People get carried away by how great AIs are at spitting out complex sentences from a huge training set that look like a human would have written them. Winter v2 is coming. Besides, there's no training set for the specific domain problem I'm currently solving. It is - by definition - not generic. The problem domain is not bubble sort or fizzbuzz.

roywiggins 4y ago
> it can never prove the code is working correctly (see halting problem). The halting problem doesn't say anything about whether a program could prove things about another program, just that it can't do it for all programs. Lots of programs obviously do or don't halt, and you can prove it in a couple lines ("there's an infinite loop here and every execution will enter it" or "all the loops are bounded by constants" etc). There's nothing stopping an AI from noticing things like that just as much as humans can. For more complex programs it obviously gets much harder, but it's very hard for humans to prove things about programs too!
- srcreigh 4y ago
  The AI can also read our proofs
- beders 4y ago
  No it doesn't, but that is essentially what a coding AI would have to do for correctness. Unless we are not talking about correctness. And we are not talking about trivial cases either.
  - roywiggins 4y ago
    An AI could correctly prove some things about some programs, and for the rest of them shrug and say it doesn't know.
codebje 4y ago
I don't think the halting problem's got anything to do with it, because I certainly also can't prove, in general, whether some code I have written halts or not either but that hasn't held me back from churning out vast quantities of text that, at first glance, might also resemble good code. But I agree with your sentiment: to get AI to write the code you want, you will need an engineer to work with it. I do expect that future AI code generation models will learn more about a specific code base over time, partially overcoming domain specific knowledge gaps.
- heavyset_go 4y ago
  You can reason why or why not you believe your code is correct, and ultimately come to an understanding if your reasoning proves correct or incorrect, but AI is just working off of patterns from its data sets.
  - sciolist 4y ago
    LLM's aren't just "matching patters", they demonstrate emergent behavior, allowing them to do much more than what they were explicitly trained on. At the moment AI models might not be able to utilize "train of thought reasoning" like we can, however the distance between these emergent behaviors and "reasoning" is much shorter than we realize (and will continue to converge in the future).
    - dinkumthinkum 4y ago
      So, do you really think these computational AI systems can reason and solve problems equivalently or in a manner superior to humans? If they aren’t matching patterns, which are I think the literature makes clear they are, why does ChatGPT fail at solving logic problems so badly? It is already performing impressive enough at tasks that people think it can replace all knowledge but yet if fails miserably at logic problems and even those that claim it replaces google do not really seems to realize it fails at providing answers to basic facts. I really don’t see any evidence that this type of AI, which is quite advanced for what it is, is type of AI that dissolves technical education / jobs. Another question, if this is on the road to being human intelligence, how is it someone like Ramanujan received no formal schooling and yet could create solutions that seem very far off from the capabilities of these systems which we are claiming will replace humans? Further, if it’s intelligence is so superior, why does it require so many watts of power produce so much thermal by-product when beings that actually do perform in a superior way currently require little more than a bowl of rice every few days to continue solving problems? Is there not sone profound, “emergent,” difference in sophistication?
olalonde 4y ago
> The author seems to forget that while an AI can create text that looks like code, it can never prove the code is working correctly (see halting problem). The halting problem applies to humans as well...
- heavyset_go 4y ago
  Humans can reason, ML models just follow statistical correlations.
  - notwokeno 4y ago
    My understanding was that it does a limited form of reasoning via "train of thought." Maybe it can only transform the ideas in the ways it was trained with but when you break it down there are only so many primitives here.
  - TravHatesMe 4y ago
    But reasoning includes statistical correlations
    - satellites 4y ago
      Statistical correlations are a subset of reasoning in general
      - ghostwriter 4y ago
        They aren't a subset of reasoning, they are data points that could be reasoned about (like whether it could be recognised as something known as a notion of <statistical correlation> and whether it make sense from a semantic point of view). Data points could be statistical or non-statistical, and it's the reasoning mind that distinguishes between the two alternatives based on other notions of the world phenomena that make statistics distinguished from non-statistics. A method of producing statistical correlations is a product of a reasoning mind and could be thought of as a subset of reasoning (if by "subset" we assume "everything produced by a reasoning mind via an act of reasoning"), but in order to recognise this "subset" another reasoning mind should firstly internalise the notions of statistics that are external phenomena to an act of reasoning itself. And "the act of reasoning" isn't proven to be "just something that produces correlations", it's more than that and nobody knows what exactly it is. Otherwise AI would be solved long time ago.
notwokeno 4y ago
While your application of the halting problem here is wrong you actually can use it to prove your point about chatGPT. Have it recite the halting problem In a new chat have it write a python program that tells you if a given string containing a python program will halt. It's great at hacking, and completely incapable of engineering.
Uehreka 4y ago
I do not share your optimism. Right now we are seeing what AI can do with one hand tied behind its back. In my experience, when Github Copilot makes mistakes they are frequently ones that it could’ve avoided if it ran static analysis on its suggestion and realized it was mentioning variable names that aren’t in my code. If the next version of Github Copilot integrates with basic VS Code tooling like Intellisense, it will be insanely powerful. Like, I see a lot of people saying things like “It’s just pattern-matching, it still needs an engineer to guide it”. That’s true of this version of Github Copilot. If it improves at the same clip as some of these other large models, I suspect it will become much much less true in a short time horizon.
- dinkumthinkum 4y ago
  If that’s true why is it so bad at solving logic puzzles?
  - Uehreka 4y ago
    It’s bad at solving logic puzzles today. We’ve just spent a decade-ish watching AI leapfrog over hurdle after hurdle that people said were going to be “impossible” or “would require reasoning/creativity” or “are at best 10 years away”. It feels foolish to assume we won’t see another unexpected leap. I would be unsurprised if GPT-4 or 5 suddenly becomes really good at math and logic problems. And long before that I expect AI to be hard to distinguish from a human in terms of day-to-day executive function stuff.
    - djmips 4y ago
      Everyone essentially doesn't know. We are but extrapolating something we don't understand to either future failure or success depending on a gut feeling. Some like you are saying to expect the unexpected while others are saying to expect this to stall out. I've even heard the openAI CEO couch his predictions behind 'assuming we don't stall out'. In this article and elsewhere it's been pointed out that how core abilities of the large language models are not well understood. Without this understanding everyone is just guessing at the future.
Existenceblinks 4y ago
[UI -> training models AI -> product] is kind of a wrong way to me. I'm old-fashioned about AI, over decade ago I was fascinated by Minimax heuristic in a game as a bot. It looked really smart I can't beat them in a deepest tree the machine can support (less pruning). My point is that the key is [UI -> "heuristic" + [business]logic -> product] is the way they should utilize these new neural network based AIs. Don't spit the product out directly because the product currently is looking like "heuristic", not a usable thing in business. Try putting logic in front and behind, especially put it far away from the final product so that the result will look like a legit magic.

by beders 4y ago

roywiggins 4y ago

> it can never prove the code is working correctly (see halting problem). The halting problem doesn't say anything about whether a program could prove things about another program, just that it can't do it for all programs. Lots of programs obviously do or don't halt, and you can prove it in a couple lines ("there's an infinite loop here and every execution will enter it" or "all the loops are bounded by constants" etc). There's nothing stopping an AI from noticing things like that just as much as humans can. For more complex programs it obviously gets much harder, but it's very hard for humans to prove things about programs too!

srcreigh 4y ago
The AI can also read our proofs
beders 4y ago
No it doesn't, but that is essentially what a coding AI would have to do for correctness. Unless we are not talking about correctness. And we are not talking about trivial cases either.
- roywiggins 4y ago
  An AI could correctly prove some things about some programs, and for the rest of them shrug and say it doesn't know.

codebje 4y ago

I don't think the halting problem's got anything to do with it, because I certainly also can't prove, in general, whether some code I have written halts or not either but that hasn't held me back from churning out vast quantities of text that, at first glance, might also resemble good code. But I agree with your sentiment: to get AI to write the code you want, you will need an engineer to work with it. I do expect that future AI code generation models will learn more about a specific code base over time, partially overcoming domain specific knowledge gaps.

heavyset_go 4y ago
You can reason why or why not you believe your code is correct, and ultimately come to an understanding if your reasoning proves correct or incorrect, but AI is just working off of patterns from its data sets.
- sciolist 4y ago
  LLM's aren't just "matching patters", they demonstrate emergent behavior, allowing them to do much more than what they were explicitly trained on. At the moment AI models might not be able to utilize "train of thought reasoning" like we can, however the distance between these emergent behaviors and "reasoning" is much shorter than we realize (and will continue to converge in the future).
  - dinkumthinkum 4y ago
    So, do you really think these computational AI systems can reason and solve problems equivalently or in a manner superior to humans? If they aren’t matching patterns, which are I think the literature makes clear they are, why does ChatGPT fail at solving logic problems so badly? It is already performing impressive enough at tasks that people think it can replace all knowledge but yet if fails miserably at logic problems and even those that claim it replaces google do not really seems to realize it fails at providing answers to basic facts. I really don’t see any evidence that this type of AI, which is quite advanced for what it is, is type of AI that dissolves technical education / jobs. Another question, if this is on the road to being human intelligence, how is it someone like Ramanujan received no formal schooling and yet could create solutions that seem very far off from the capabilities of these systems which we are claiming will replace humans? Further, if it’s intelligence is so superior, why does it require so many watts of power produce so much thermal by-product when beings that actually do perform in a superior way currently require little more than a bowl of rice every few days to continue solving problems? Is there not sone profound, “emergent,” difference in sophistication?

olalonde 4y ago

> The author seems to forget that while an AI can create text that looks like code, it can never prove the code is working correctly (see halting problem). The halting problem applies to humans as well...

heavyset_go 4y ago
Humans can reason, ML models just follow statistical correlations.
- notwokeno 4y ago
  My understanding was that it does a limited form of reasoning via "train of thought." Maybe it can only transform the ideas in the ways it was trained with but when you break it down there are only so many primitives here.
- TravHatesMe 4y ago
  But reasoning includes statistical correlations
  - satellites 4y ago
    Statistical correlations are a subset of reasoning in general
    - ghostwriter 4y ago
      They aren't a subset of reasoning, they are data points that could be reasoned about (like whether it could be recognised as something known as a notion of <statistical correlation> and whether it make sense from a semantic point of view). Data points could be statistical or non-statistical, and it's the reasoning mind that distinguishes between the two alternatives based on other notions of the world phenomena that make statistics distinguished from non-statistics. A method of producing statistical correlations is a product of a reasoning mind and could be thought of as a subset of reasoning (if by "subset" we assume "everything produced by a reasoning mind via an act of reasoning"), but in order to recognise this "subset" another reasoning mind should firstly internalise the notions of statistics that are external phenomena to an act of reasoning itself. And "the act of reasoning" isn't proven to be "just something that produces correlations", it's more than that and nobody knows what exactly it is. Otherwise AI would be solved long time ago.

notwokeno 4y ago

While your application of the halting problem here is wrong you actually can use it to prove your point about chatGPT. Have it recite the halting problem In a new chat have it write a python program that tells you if a given string containing a python program will halt. It's great at hacking, and completely incapable of engineering.

Uehreka 4y ago

I do not share your optimism. Right now we are seeing what AI can do with one hand tied behind its back. In my experience, when Github Copilot makes mistakes they are frequently ones that it could’ve avoided if it ran static analysis on its suggestion and realized it was mentioning variable names that aren’t in my code. If the next version of Github Copilot integrates with basic VS Code tooling like Intellisense, it will be insanely powerful. Like, I see a lot of people saying things like “It’s just pattern-matching, it still needs an engineer to guide it”. That’s true of this version of Github Copilot. If it improves at the same clip as some of these other large models, I suspect it will become much much less true in a short time horizon.

dinkumthinkum 4y ago
If that’s true why is it so bad at solving logic puzzles?
- Uehreka 4y ago
  It’s bad at solving logic puzzles today. We’ve just spent a decade-ish watching AI leapfrog over hurdle after hurdle that people said were going to be “impossible” or “would require reasoning/creativity” or “are at best 10 years away”. It feels foolish to assume we won’t see another unexpected leap. I would be unsurprised if GPT-4 or 5 suddenly becomes really good at math and logic problems. And long before that I expect AI to be hard to distinguish from a human in terms of day-to-day executive function stuff.
  - djmips 4y ago
    Everyone essentially doesn't know. We are but extrapolating something we don't understand to either future failure or success depending on a gut feeling. Some like you are saying to expect the unexpected while others are saying to expect this to stall out. I've even heard the openAI CEO couch his predictions behind 'assuming we don't stall out'. In this article and elsewhere it's been pointed out that how core abilities of the large language models are not well understood. Without this understanding everyone is just guessing at the future.

Existenceblinks 4y ago

[UI -> training models AI -> product] is kind of a wrong way to me. I'm old-fashioned about AI, over decade ago I was fascinated by Minimax heuristic in a game as a bot. It looked really smart I can't beat them in a deepest tree the machine can support (less pruning). My point is that the key is [UI -> "heuristic" + [business]logic -> product] is the way they should utilize these new neural network based AIs. Don't spit the product out directly because the product currently is looking like "heuristic", not a usable thing in business. Try putting logic in front and behind, especially put it far away from the final product so that the result will look like a legit magic.