4 ms·

Ask it to write a backtracking sudoku solver with coroutines and/or fibers and let me know how it performs in your language of choice. We are nowhere near gene

Ask it to write a backtracking sudoku solver with coroutines and/or fibers and let me know how it performs in your language of choice. We are nowhere near generally intelligent software systems.

jtmoulia 3y ago
I was curious where GPT-4 would come up short on the problem and I was surprised -- it seemed to solve it pretty well whether or not using coroutines. (I dropped both solns into a python interpreter and both appeared to solve the problem.) There could def be bugs I missed tho. https://chat.openai.com/share/ef77507e-cb75-4112-97f1-a16cfc03cd98 https://chat.openai.com/share/ef77507e-cb75-4112-97f1-a16cfc...
- climatologist 3y ago
  That's a good attempt but the coroutine solution is incorrect. See if you can figure out why and how to improve it. You can also ask it to propagate constraints and see what happens.
  - el_nahual 3y ago
    This isn't the dunk you think it is since GP is a human (I presume), and he thought the solution worked.
  - jtmoulia 3y ago
    Sigh classic LLM -- without you, the expert, I can't quickly tell from the code / output how the answer the LLM produced is wrong. I also asked it to solve sudoku by "propagating constraints" and the answer seemed to work for me :/ Again, I'd guess the soln produced is wrong because I trust you more than the LLM but I don't have the mental horsepower to figure it out without resorting to tests & debugging.
    - climatologist 3y ago
      If you are happy with the tool then continue using it. My point was a simple one, any generally intelligent software system would have given you an optimal solution by accounting for what people know about backtracking and constraint propagation but the solution presented did not account for how to optimally use coroutines to propagate constraints. I'll repeat what I said previously in a different way, LLMs are useful but they are nowhere near what is required to achieve generally intelligent software systems. I'm sure they will continue to improve as engineers and companies learn how to utilize them in their workflows but let's temper the hype a little bit because statistical autocompletion is not enough to achieve general intelligence.
_gabe_ 3y ago
Ask a random college freshman in computer science to write a backtracking sudoku solver with coroutines and/or fibers in their language of choice. Oof, only 1 out of 10 can do it...? I guess 9 out of 10 college freshmen aren't generally intelligent? (I mean, I certainly wasn't very intelligent as a college freshman, but I'm pretty sure that's not the way you're using that word here). Anyways, these kinds of strawmen always baffle me in regards to AI. Insert random pseudo trivia that most of the general population wouldn't be able to do, see the AI fail at that specific task. "What did I tell you? The AI definitely isn't generally intelligent yet!" Everybody is out to prove AI isn't intelligent without first defining what intelligence even is. And when other people rightly point out it can do a lot of stuff, they point to some specific task that it gets 90% of the way there but doesn't get perfect and then triumphantly declare AI isn't intelligent. Crazy.
- climatologist 3y ago
  I think everyone should have a test in their back pocket for testing claims made by companies making AI tools. Mine is backtracking sudoku solver with constraint propagation but that's because I know LLMs are incapable of recursion and backtracking. Usually people get the point I'm making but sometimes I have to elaborate further. LLMs are cool toys but calling them intelligent is stretching the definition of "intelligent" way too far. It's important to be clear about what the words actually mean because if people start thinking these software systems can be substituted for their own thinking then we end up with all sorts of unnecessary confusion around what they're actually capable of achieving.
  - famouswaffles 3y ago
    Any test a lot of humans will fail is simply not a test of General Intelligence. That should be obvious enough. And GPT-4 does do this in a couple iterations.
    - climatologist 3y ago
      Humans are not generally intelligent.
pulvinar 3y ago
If you're saying it's not perfect, then sure: it required one iteration before the Python program it spat out worked. It also when asked gave a 2-page description of the code. And how many humans can do this? Those poor exhausted goalposts.
IanCal 3y ago
I have strongly believed the best way of evaluating the state of AI is to look at what people complain it can't do. It's taken a few short years to go from "keep a coherent story over more than a sentence" to "write a multithreaded sudoku solver in one shot". Many programmers would fail to do this. I'm not even sure a randomly selected human would understand the question. And I'm wondering if you've setup any loops letting it write tests, search the internet, inspect and debug? If not what you see as an output is essentially it whiteboarding off the top of its "head". Programmers routinely fail to solve simpler things in interviews, and their skills are much narrower than current llms (you can easily argue deeper, but I feel comfortable saying broader).
- climatologist 3y ago
  > write a multithreaded sudoku solver in one shot I did not say anything about threads but I did hint at the fact that coroutines can be used creatively to solve problems which require backtracking and constraint propagation. The fact that this is still controversial means a lot of people are unaware LLMs do not support backtracking, recursion, and constraint propagation. Sudoku is just an obvious example of a problem that is easy to solve if you know about these concepts and next to impossible if you don't. Such problems can also be expressed as integer programs but even fewer people know how to do that so I don't usually bring it up but it would be another good test for any software system that is claimed to be intelligent by the corporate marketing department.
  - IanCal 3y ago
    Llms don't have to support something internally to know about those concepts, those are entirely different things. > Such problems can also be expressed as integer programs but even fewer people know how to do that so I don't usually bring it up but it would be another good test for any software system that is claimed to be intelligent by the corporate marketing department. It's very telling if the level of testing is "can it do something few people in the field can?".
    - climatologist 3y ago

Ask it to write a backtracking sudoku solver with coroutines and/or fibers and let me know how it performs in your language of choice. We are nowhere near generally intelligent software systems.

jtmoulia 3y ago

I was curious where GPT-4 would come up short on the problem and I was surprised -- it seemed to solve it pretty well whether or not using coroutines. (I dropped both solns into a python interpreter and both appeared to solve the problem.) There could def be bugs I missed tho. https://chat.openai.com/share/ef77507e-cb75-4112-97f1-a16cfc03cd98 https://chat.openai.com/share/ef77507e-cb75-4112-97f1-a16cfc...

climatologist 3y ago
That's a good attempt but the coroutine solution is incorrect. See if you can figure out why and how to improve it. You can also ask it to propagate constraints and see what happens.
- el_nahual 3y ago
  This isn't the dunk you think it is since GP is a human (I presume), and he thought the solution worked.
- jtmoulia 3y ago
  Sigh classic LLM -- without you, the expert, I can't quickly tell from the code / output how the answer the LLM produced is wrong. I also asked it to solve sudoku by "propagating constraints" and the answer seemed to work for me :/ Again, I'd guess the soln produced is wrong because I trust you more than the LLM but I don't have the mental horsepower to figure it out without resorting to tests & debugging.
  - climatologist 3y ago
    If you are happy with the tool then continue using it. My point was a simple one, any generally intelligent software system would have given you an optimal solution by accounting for what people know about backtracking and constraint propagation but the solution presented did not account for how to optimally use coroutines to propagate constraints. I'll repeat what I said previously in a different way, LLMs are useful but they are nowhere near what is required to achieve generally intelligent software systems. I'm sure they will continue to improve as engineers and companies learn how to utilize them in their workflows but let's temper the hype a little bit because statistical autocompletion is not enough to achieve general intelligence.

_gabe_ 3y ago

Ask a random college freshman in computer science to write a backtracking sudoku solver with coroutines and/or fibers in their language of choice. Oof, only 1 out of 10 can do it...? I guess 9 out of 10 college freshmen aren't generally intelligent? (I mean, I certainly wasn't very intelligent as a college freshman, but I'm pretty sure that's not the way you're using that word here). Anyways, these kinds of strawmen always baffle me in regards to AI. Insert random pseudo trivia that most of the general population wouldn't be able to do, see the AI fail at that specific task. "What did I tell you? The AI definitely isn't generally intelligent yet!" Everybody is out to prove AI isn't intelligent without first defining what intelligence even is. And when other people rightly point out it can do a lot of stuff, they point to some specific task that it gets 90% of the way there but doesn't get perfect and then triumphantly declare AI isn't intelligent. Crazy.

climatologist 3y ago
I think everyone should have a test in their back pocket for testing claims made by companies making AI tools. Mine is backtracking sudoku solver with constraint propagation but that's because I know LLMs are incapable of recursion and backtracking. Usually people get the point I'm making but sometimes I have to elaborate further. LLMs are cool toys but calling them intelligent is stretching the definition of "intelligent" way too far. It's important to be clear about what the words actually mean because if people start thinking these software systems can be substituted for their own thinking then we end up with all sorts of unnecessary confusion around what they're actually capable of achieving.
- famouswaffles 3y ago
  Any test a lot of humans will fail is simply not a test of General Intelligence. That should be obvious enough. And GPT-4 does do this in a couple iterations.
  - climatologist 3y ago
    Humans are not generally intelligent.

pulvinar 3y ago

If you're saying it's not perfect, then sure: it required one iteration before the Python program it spat out worked. It also when asked gave a 2-page description of the code. And how many humans can do this? Those poor exhausted goalposts.

IanCal 3y ago

I have strongly believed the best way of evaluating the state of AI is to look at what people complain it can't do. It's taken a few short years to go from "keep a coherent story over more than a sentence" to "write a multithreaded sudoku solver in one shot". Many programmers would fail to do this. I'm not even sure a randomly selected human would understand the question. And I'm wondering if you've setup any loops letting it write tests, search the internet, inspect and debug? If not what you see as an output is essentially it whiteboarding off the top of its "head". Programmers routinely fail to solve simpler things in interviews, and their skills are much narrower than current llms (you can easily argue deeper, but I feel comfortable saying broader).

climatologist 3y ago
> write a multithreaded sudoku solver in one shot I did not say anything about threads but I did hint at the fact that coroutines can be used creatively to solve problems which require backtracking and constraint propagation. The fact that this is still controversial means a lot of people are unaware LLMs do not support backtracking, recursion, and constraint propagation. Sudoku is just an obvious example of a problem that is easy to solve if you know about these concepts and next to impossible if you don't. Such problems can also be expressed as integer programs but even fewer people know how to do that so I don't usually bring it up but it would be another good test for any software system that is claimed to be intelligent by the corporate marketing department.
- IanCal 3y ago
  Llms don't have to support something internally to know about those concepts, those are entirely different things. > Such problems can also be expressed as integer programs but even fewer people know how to do that so I don't usually bring it up but it would be another good test for any software system that is claimed to be intelligent by the corporate marketing department. It's very telling if the level of testing is "can it do something few people in the field can?".
  - climatologist 3y ago