6 ms·

I think it must have seen the solution somewhere already on the web. I find it extremely hard to believe that such a general purpose chatbot would just be able

I think it must have seen the solution somewhere already on the web. I find it extremely hard to believe that such a general purpose chatbot would just be able to solve programming problems. Deepmind had a paper [1] on solving programming problems a couple of months ago and they had to apply quite specialized heuristics in order to solve these problems. Obviously ChatGPT does nothing of the sorts and it just seems extremely unrealistic that it would be capable of outperforming previous work like that. [1] https://news.ycombinator.com/item?id=30179549 https://news.ycombinator.com/item?id=30179549

ZiiS 4y ago
This was the first solution, submitted very quickly after the problem was published.
- gregwebs 4y ago
  For one problem. For the rest of the problems it has been very challenging to get the AI to write the correct solution. Still an impressive result that with specification, testing, and feedback the AI can come up with the correct result in the end.
Waterluvian 4y ago
I’ve had this skepticism until I started asking the bot to do things like “create a limerick using some words that have never existed before and then explain the definition of those words.” I don’t believe the AI is generalized by any means. But I think it’s more than a Eliza-style parlour trick. It does really well with facts but misses subjectivity and nuance. Like I ask it to write stories for Seinfeld or Raymond and it misses the tone and style entirely. But it gets all the names and other factual details right. I find it works incredibly well when you want things that are objective, like code or rhymes or comparisons of things.
- FiberBundle 4y ago
  What does any of this have to do with solving programming problems?
- tasuki 4y ago
  > I’ve had this skepticism until I started asking the bot to do things like “create a limerick using some words that have never existed before and then explain the definition of those words.” Sounds interesting! Mind sharing the results?
  - Waterluvian 4y ago
    I’m not saying they’re good. Just that they’re (mostly) following the rules and aren’t just thematically random. https://ibb.co/RpSxcV8 https://ibb.co/RpSxcV8 https://ibb.co/883GpHY https://ibb.co/883GpHY
    - jwuphysics 4y ago
      That first one is pretty good.
    - tasuki 4y ago
      Perhaps they aren't good, but they aren't that bad either. Probably better than I'd manage...
- aew4ytasghe5 4y ago
  > things that are objective, like code or rhymes rappers disagree
  - Waterluvian 4y ago
    Oh my yes, for sure. I mean the literary rules of a rhyme or a limerick. Rules that artists can, do, and should break for effect.
- tartoran 4y ago
  Though it may look amazing it is not really thinking. It is simply capable of using a placeholder word. For HN’s sake AI does not understand anything, it’s just leveraging patterns, patterns that are sometimes complicated for us and that does make a great tool. But it’s just a tool for now.
  - Waterluvian 4y ago
    We are all philosophical zombies. Let’s not single out the software. ;)
  - baandam 4y ago
    AlphaZero and Stockfish are not really thinking either when it comes to chess. I just don't understand how a thinking human can not see that this is basically irrelevant.
  - motoxpro 4y ago
    Take a look at the sibling. Maybe be less than understanding but it’s a lot more than a placeholder.
- hansbo 4y ago
  It's really quite amazing. Inspired by your example, I tried to make it create something which nobody could have thought about before: Write a deep poem about life, death, and the website Rotten Tomatoes The result: Life is a fleeting flame A dance in the endless game We come and we go Leaving our mark, but never to know What the future holds or who's to blame Death is the great unknown A mystery, never to be shown But Rotten Tomatoes, a website so bold Attempts to rate our stories untold As if life and death can be bought and sold. But in the end, what does it matter? The ratings, the reviews, the bitter chatter For in the grand scheme of things We are but fleeting moments, the faintest of flings And Rotten Tomatoes, a mere blip on the radar.
logicallee 4y ago
You can check this by taking some time to create some specific weird puzzle that is unlikely to have been made in the format you come up with, then see if it can solve it. If you don't write it anywhere then it is being solved for the first time. Just make sure it is a pretty unusual puzzle.
frontman1988 4y ago
Yeah it can't solve novel questions. The AI couldn't do much to solve this one for example: https://codeforces.com/contest/1672/problem/D https://codeforces.com/contest/1672/problem/D ChatGPT is just a very good copy/paste, not a logical problem solver(yet).
- dislikedtom2 4y ago
  To be fair, humans also have great problems solving novel problems. Monkey see, monkey do!
- jcims 4y ago
  Have you tried describing the algorithm? https://imgur.com/a/7da1vFj https://imgur.com/a/7da1vFj I used 'z' instead of 'a' to avoid any possible issues with the article 'a'. I think I messed up the assignment of z[l] at z[r] being after its updated, not sure. But it created the input format, described it and the program ran the first time once i fixed the indents (code formatting is broken for some reason). If I run against the input at the contest page I get NO NO NO NO YES.
  - pedrosorio 4y ago
    Isn't the expected output YES YES NO YES NO?
    - jcims 4y ago
      Yes according to the puzzle, but the operation as described isn’t clear (to me anyway) about which value should be used for the final assignment. It just says to use the value from the first element, but it’s unclear if it’s before or after that first element is replaced by the first operation.
      - pedrosorio 4y ago
        I see, your description to GPT does not match the problem statement. > It just says to use the value from the first element, but it’s unclear if it’s before or after that first element is replaced by the first operation. Is the following python code unclear to you? a = 0 b = 1 a, b = b, a or alternatively: a = [1,2,3,4,5] a[0:3] = a[1:3] + [a[0]] There's nothing in the statement that indicates the assignments should be done one element at a time (if so, the order of the assignments would need to be specified). It's an atomic operation that circularly shifts the values in the array in the range l...r
12345hn6789 4y ago
These first few advent problems are extremely trivial. Solvable <1 min with experience programmers. And are at the level of someone with cs 101 knowledge. Personally I don't see it being difficult for the AI to solve these trivial problems at all.
pps 4y ago
Does it have access to the web? For one of my questions it answered: "Unfortunately, I am unable to provide a detailed description of the education system in Poland and its changes over the last 30 years because I have limited access to information and cannot browse the internet." but I have no idea if it's not lying :)
- swid 4y ago
  Some posts yesterday showed that network is usually disabled, but with the right prompt, you can enable network and someone got it to like a Twitter post.
  - OctopusLupid 4y ago
    I think the “liking Twitter post” part was just a coincidental joke.
    - iudqnolq 4y ago
      Correct. He also claimed the chatbot signed in as Grime's Twitter account to do the action, which is obviously implausible. If you look in the replies you'll see him clarify it was a joke.
    - swid 4y ago
      Thanks for the correction, guess I did not read that post close enough.
  - pps 4y ago
    I would like to see how to achieve that. I tried asking it to translate the first paragraph of a site, but it provided more of an interpretation or summary of the article, rather than the actual text. When asked to copy the text from the website, it said that it couldn't. Additionally, when asked to provide a summary of a website that didn't exist before (created it myself yesterday), it gave a summary that was completely fictional, based on its interpretation of the URL.
MatthiasPortzel 4y ago
The top leaderboard spots for AoC day 2 were taken by people who passed the response directly into GTP3. https://twitter.com/max_sixty/status/1598924237947154433 https://twitter.com/max_sixty/status/1598924237947154433 The AoC challenges this early aren't difficult, but they have several steps and are significantly more challenging than something you would be able to find as a Stackoverflow answer.
joaogui1 4y ago
AlphaCode is solving much harder problems than these first few days of AoC

ZiiS 4y ago

This was the first solution, submitted very quickly after the problem was published.

gregwebs 4y ago
For one problem. For the rest of the problems it has been very challenging to get the AI to write the correct solution. Still an impressive result that with specification, testing, and feedback the AI can come up with the correct result in the end.

Waterluvian 4y ago

I’ve had this skepticism until I started asking the bot to do things like “create a limerick using some words that have never existed before and then explain the definition of those words.” I don’t believe the AI is generalized by any means. But I think it’s more than a Eliza-style parlour trick. It does really well with facts but misses subjectivity and nuance. Like I ask it to write stories for Seinfeld or Raymond and it misses the tone and style entirely. But it gets all the names and other factual details right. I find it works incredibly well when you want things that are objective, like code or rhymes or comparisons of things.

FiberBundle 4y ago
What does any of this have to do with solving programming problems?
tasuki 4y ago
> I’ve had this skepticism until I started asking the bot to do things like “create a limerick using some words that have never existed before and then explain the definition of those words.” Sounds interesting! Mind sharing the results?
- Waterluvian 4y ago
  I’m not saying they’re good. Just that they’re (mostly) following the rules and aren’t just thematically random. https://ibb.co/RpSxcV8 https://ibb.co/RpSxcV8 https://ibb.co/883GpHY https://ibb.co/883GpHY
  - jwuphysics 4y ago
    That first one is pretty good.
  - tasuki 4y ago
    Perhaps they aren't good, but they aren't that bad either. Probably better than I'd manage...
aew4ytasghe5 4y ago
> things that are objective, like code or rhymes rappers disagree
- Waterluvian 4y ago
  Oh my yes, for sure. I mean the literary rules of a rhyme or a limerick. Rules that artists can, do, and should break for effect.
tartoran 4y ago
Though it may look amazing it is not really thinking. It is simply capable of using a placeholder word. For HN’s sake AI does not understand anything, it’s just leveraging patterns, patterns that are sometimes complicated for us and that does make a great tool. But it’s just a tool for now.
- Waterluvian 4y ago
  We are all philosophical zombies. Let’s not single out the software. ;)
- baandam 4y ago
  AlphaZero and Stockfish are not really thinking either when it comes to chess. I just don't understand how a thinking human can not see that this is basically irrelevant.
- motoxpro 4y ago
  Take a look at the sibling. Maybe be less than understanding but it’s a lot more than a placeholder.
hansbo 4y ago
It's really quite amazing. Inspired by your example, I tried to make it create something which nobody could have thought about before: Write a deep poem about life, death, and the website Rotten Tomatoes The result: Life is a fleeting flame A dance in the endless game We come and we go Leaving our mark, but never to know What the future holds or who's to blame Death is the great unknown A mystery, never to be shown But Rotten Tomatoes, a website so bold Attempts to rate our stories untold As if life and death can be bought and sold. But in the end, what does it matter? The ratings, the reviews, the bitter chatter For in the grand scheme of things We are but fleeting moments, the faintest of flings And Rotten Tomatoes, a mere blip on the radar.

logicallee 4y ago

You can check this by taking some time to create some specific weird puzzle that is unlikely to have been made in the format you come up with, then see if it can solve it. If you don't write it anywhere then it is being solved for the first time. Just make sure it is a pretty unusual puzzle.

frontman1988 4y ago

Yeah it can't solve novel questions. The AI couldn't do much to solve this one for example: https://codeforces.com/contest/1672/problem/D https://codeforces.com/contest/1672/problem/D ChatGPT is just a very good copy/paste, not a logical problem solver(yet).

dislikedtom2 4y ago
To be fair, humans also have great problems solving novel problems. Monkey see, monkey do!
jcims 4y ago
Have you tried describing the algorithm? https://imgur.com/a/7da1vFj https://imgur.com/a/7da1vFj I used 'z' instead of 'a' to avoid any possible issues with the article 'a'. I think I messed up the assignment of z[l] at z[r] being after its updated, not sure. But it created the input format, described it and the program ran the first time once i fixed the indents (code formatting is broken for some reason). If I run against the input at the contest page I get NO NO NO NO YES.
- pedrosorio 4y ago
  Isn't the expected output YES YES NO YES NO?
  - jcims 4y ago
    Yes according to the puzzle, but the operation as described isn’t clear (to me anyway) about which value should be used for the final assignment. It just says to use the value from the first element, but it’s unclear if it’s before or after that first element is replaced by the first operation.
    - pedrosorio 4y ago
      I see, your description to GPT does not match the problem statement. > It just says to use the value from the first element, but it’s unclear if it’s before or after that first element is replaced by the first operation. Is the following python code unclear to you? a = 0 b = 1 a, b = b, a or alternatively: a = [1,2,3,4,5] a[0:3] = a[1:3] + [a[0]] There's nothing in the statement that indicates the assignments should be done one element at a time (if so, the order of the assignments would need to be specified). It's an atomic operation that circularly shifts the values in the array in the range l...r

12345hn6789 4y ago

These first few advent problems are extremely trivial. Solvable <1 min with experience programmers. And are at the level of someone with cs 101 knowledge. Personally I don't see it being difficult for the AI to solve these trivial problems at all.

pps 4y ago

Does it have access to the web? For one of my questions it answered: "Unfortunately, I am unable to provide a detailed description of the education system in Poland and its changes over the last 30 years because I have limited access to information and cannot browse the internet." but I have no idea if it's not lying :)

swid 4y ago
Some posts yesterday showed that network is usually disabled, but with the right prompt, you can enable network and someone got it to like a Twitter post.
- OctopusLupid 4y ago
  I think the “liking Twitter post” part was just a coincidental joke.
  - iudqnolq 4y ago
    Correct. He also claimed the chatbot signed in as Grime's Twitter account to do the action, which is obviously implausible. If you look in the replies you'll see him clarify it was a joke.
  - swid 4y ago
    Thanks for the correction, guess I did not read that post close enough.
- pps 4y ago
  I would like to see how to achieve that. I tried asking it to translate the first paragraph of a site, but it provided more of an interpretation or summary of the article, rather than the actual text. When asked to copy the text from the website, it said that it couldn't. Additionally, when asked to provide a summary of a website that didn't exist before (created it myself yesterday), it gave a summary that was completely fictional, based on its interpretation of the URL.

MatthiasPortzel 4y ago

The top leaderboard spots for AoC day 2 were taken by people who passed the response directly into GTP3. https://twitter.com/max_sixty/status/1598924237947154433 https://twitter.com/max_sixty/status/1598924237947154433 The AoC challenges this early aren't difficult, but they have several steps and are significantly more challenging than something you would be able to find as a Stackoverflow answer.

joaogui1 4y ago

AlphaCode is solving much harder problems than these first few days of AoC