6 ms·

Ye I signed up and tried some queries. It was quite scary. "Write a function that sorts the elements of a c string backwards, in C." "Add a flag to the sort fu

Ye I signed up and tried some queries. It was quite scary. "Write a function that sorts the elements of a c string backwards, in C." "Add a flag to the sort function which makes it sort the string forward." "Could you write a endian swap function for double argument in PIC assembler?" "Could you write a binary sort in PIC 8-bit assembler" "Write a Javascript function that prints the day of the week together with some Chinese words of wisdom." It had no problem doing any one those. I ran them all, except the assembler ones. The question is how good it is to process larger chunks of code and makes changes to it. People thinking about becoming programmers might need to rethink their plans if this one improves ... EDIT: Oh dear. I introduced bugs in its sort code and it found them and explained what they did.

guiambros 4y ago
Last night I entered the page-long instructions for Advent of Code day 4, and it spewed out perfectly readable code and solved it on the first try [1]. And we're not talking about a common algorithm that has been solved many times before, but a convoluted story that is full of "elves cleaning up overlapping sections of the camp" (!), and ChatGPT was still able to understand it, write the code to solve it, and even explain how it works. It's nothing short of a phenomenal milestone. [1] https://twitter.com/GuiAmbros/status/1599282083838296064 https://twitter.com/GuiAmbros/status/1599282083838296064
- ak_111 4y ago
  Are you sure this whole problem with its solution wasn't indexed in ChatGPT training set, so all it did here is retrieve the whole answer to the problem similar to how google will find the solution pretty quickly if you search for "advent of code day 4 solution"? Can you see if it solves an advent of code problem whose solution hasn't been posted yet? If it did I will consider this an almost publishable breakthrough.
  - ghusbands 4y ago
    It wasn't in the training set - the puzzle was released today, as part of this year's advent of code. ChatGPT is eminently publishable and has demonstrably produced plenty of novel, if boring, code.
    - TedDoesntTalk 4y ago
      Genuine question: can it write larger codebases yet such as: “write a pac-man clone in JavaScript, html, and css”
      - negamax 4y ago
        It can't write pacman. But if you describe a scenario to it, it writes pretty accurate code and can drill down. Very impressive
      - MacsHeadroom 4y ago
        Yes, but you'll have to chop the project up into different parts and describe each sufficiently well. For example, start with "write a level generator in JavaScript which outputs pacman like levels encoded in ASCII which could be used as the basis for a pacman clone." Then you'd say "Great, now write a pacman clone in JavaScript which takes the input of the previously generated level generator and takes interactive input on the console." Then you'd have to ask it to generate the HTML and CSS. It won't do it all at once. But it will do it piece by piece.
        rafabulsing 4y ago
        Yes, it's absolutely feasible. I did exactly that with a simpler version of this (Tic Tac Toe, and console only, no actual visual interface) but it worked pretty well. The main thing is having a good idea of what the overall program structure will look like. If you have that, I think you can go pretty far with it. Btw, in case anyone wants to see what working like this may look like, I recorded myself while doing that little experiment: https://youtu.be/TBMQIDMwI5E https://youtu.be/TBMQIDMwI5E
      - dwild 4y ago
  - ignu 4y ago
    I've tried giving it languages like Turbo Pascal or telling it to obscure frameworks like fp-ts and while sometimes it's not syntactically perfect, but I can guarntee it's not just scraping code snippets from somewhere.
  - dwild 4y ago
    The guy that got the first position for the first part, thus the first solution ever on that problem, was using ChatGPT to do it. [1] [1] https://github.com/max-sixty/aoc-gpt https://github.com/max-sixty/aoc-gpt
- deleted 4y ago
  [deleted]
alexeldeib 4y ago
I tried higher level problems: write an http server in rust using warp and postgres, with endpoint for get users and put user. 9 compile errors, some due to cargo.toml issues (it only generated main.rs). But mostly worked. The post we’re discussing is also pretty impressive, a working lexer/parser
nwienert 4y ago
The answer is it’s very bad at large amounts of code. It has no skill at reasoning across a large amount of files where say you’d need to make many small changes in a handful of places and know that those changes all work together. These models are amazing at regurgitation. They have no skill at reasoning and any solution outside their training set. They have a distinct disability for precision in anything remotely novel or large. And they will bullshit subtle bugs all over the place at scale. Even a junior programmer would produce more value than the next generations of this, simply because they can take vague English instructions and actually make iterative progress across a huge codebase, test those changes for all sorts of vague constraints, and be precise. There’s no ability for that here.
- passion__desire 4y ago
  This tool would make for a great pair programmer and an intelligent interactive rubber duck
  - birdyrooster 4y ago
    Except you must have the underlying knowledge to effectively review code. So how that would obviate a career in programming is a strange choice by the grandparent commenter. Anyone can rubber stamp bad code, but to stop bad code from entering projects you need experts.
    - aprdm 4y ago
      Most projects have plenty of bad code, nobody cares, as long as it solves the problem.
      - ad404b8a372f2b9 4y ago
        The bad code it produces is not just stylistically bad, it often doesn't solve the problem. It seems to be making the same kind of errors in code as we've seen in speech, you get a lot of plausible but subtly wrong hallucinated content that only someone who knows the subject can identify. It prefers probable output over matching the specifications so it'll add unnecessary branches just because it's seen them millions of times. Among errors I've seen it make there is bad indexing, adding conditions that serve no purpose or assuming a variable has a specific structure when it doesn't.
xur17 4y ago
> The question is how good it is to process larger chunks of code and makes changes to it. > People thinking about becoming programmers might need to rethink their plans if this one improves ... Very true, and this thought definitely crossed my mind as well. In the short term I imagine this is going to be like any other tool, it will increase your leverage, and make it possible for you to write code faster. For example, I fed it 100 lines of a Django model and asked it to write some query code for me. Produced something that was possible faster than I could have. The few slight deficiencies it did have (optimizations) basically felt like me doing a code review.
dopidopHN 4y ago
Hi, thanks for breaking it down. Do you write code close to the machine at $job? I wanna try with more feature and UI heavy request. I actually comment because of your edit in your response : you manually introduced bugs, reran the process, and it seen you broke it ( fine ) but was able to explain it to you in English ? Where is that comming from ? Adjacent comments and git comments?
- makestuff 4y ago
  I asked it to make some stuff in flutter and it worked well. I did notice sometimes it just adds a comment in the function saying “insert logic here for X” when it doesn’t know how to write it.
- rightbyte 4y ago
  > I actually comment because of your edit in your response : you manually introduced bugs, reran the process, and it seen you broke it ( fine ) but was able to explain it to you in English ? I fed it it's own sorting function but I added bugs to it and asked it to spot the bugs.
sage76 4y ago
I tried a hard level leetcode problem on it. Perfect results on the first attempt.
- dvngnt_ 4y ago
  it was trained with leetcode data though?
  - sage76 4y ago
    Ah I didn't know that.
deleted 4y ago
[deleted]
nobody5050 4y ago
Something I’ve noticed in my own experimentations is that it’s really only limited by the length of its replies. It can happily generate perfect code but then stop in the middle of a function simply because the website only asked for 200 tokens or something. It’s really quite something to say that one of my gripes lies not in the model but in the user interface wrapping the model.
melagonster 4y ago
programmers are final worker. there are not no more jobs.
ChickeNES 4y ago
I already managed to get it to write an implementation of Pong and a Chip8 interpreter, both in C and using SDL2 for graphics/input/sound (it even wrote a function to generate the waveform for a beep for Chip8 and rebound sound for Pong!). I'm pretty sure I can get it to generate all the code for a Gameboy or NES emulator, I've taken a few tentative steps and it definitely can reproduce the memory map and generate code to parse iNES/GB cartridge headers

guiambros 4y ago

Last night I entered the page-long instructions for Advent of Code day 4, and it spewed out perfectly readable code and solved it on the first try [1]. And we're not talking about a common algorithm that has been solved many times before, but a convoluted story that is full of "elves cleaning up overlapping sections of the camp" (!), and ChatGPT was still able to understand it, write the code to solve it, and even explain how it works. It's nothing short of a phenomenal milestone. [1] https://twitter.com/GuiAmbros/status/1599282083838296064 https://twitter.com/GuiAmbros/status/1599282083838296064

ak_111 4y ago
Are you sure this whole problem with its solution wasn't indexed in ChatGPT training set, so all it did here is retrieve the whole answer to the problem similar to how google will find the solution pretty quickly if you search for "advent of code day 4 solution"? Can you see if it solves an advent of code problem whose solution hasn't been posted yet? If it did I will consider this an almost publishable breakthrough.
- ghusbands 4y ago
  It wasn't in the training set - the puzzle was released today, as part of this year's advent of code. ChatGPT is eminently publishable and has demonstrably produced plenty of novel, if boring, code.
  - TedDoesntTalk 4y ago
    Genuine question: can it write larger codebases yet such as: “write a pac-man clone in JavaScript, html, and css”
    - negamax 4y ago
      It can't write pacman. But if you describe a scenario to it, it writes pretty accurate code and can drill down. Very impressive
    - MacsHeadroom 4y ago
      Yes, but you'll have to chop the project up into different parts and describe each sufficiently well. For example, start with "write a level generator in JavaScript which outputs pacman like levels encoded in ASCII which could be used as the basis for a pacman clone." Then you'd say "Great, now write a pacman clone in JavaScript which takes the input of the previously generated level generator and takes interactive input on the console." Then you'd have to ask it to generate the HTML and CSS. It won't do it all at once. But it will do it piece by piece.
      - rafabulsing 4y ago
        Yes, it's absolutely feasible. I did exactly that with a simpler version of this (Tic Tac Toe, and console only, no actual visual interface) but it worked pretty well. The main thing is having a good idea of what the overall program structure will look like. If you have that, I think you can go pretty far with it. Btw, in case anyone wants to see what working like this may look like, I recorded myself while doing that little experiment: https://youtu.be/TBMQIDMwI5E https://youtu.be/TBMQIDMwI5E
    - dwild 4y ago
- ignu 4y ago
  I've tried giving it languages like Turbo Pascal or telling it to obscure frameworks like fp-ts and while sometimes it's not syntactically perfect, but I can guarntee it's not just scraping code snippets from somewhere.
- dwild 4y ago
  The guy that got the first position for the first part, thus the first solution ever on that problem, was using ChatGPT to do it. [1] [1] https://github.com/max-sixty/aoc-gpt https://github.com/max-sixty/aoc-gpt
deleted 4y ago
[deleted]

alexeldeib 4y ago

I tried higher level problems: write an http server in rust using warp and postgres, with endpoint for get users and put user. 9 compile errors, some due to cargo.toml issues (it only generated main.rs). But mostly worked. The post we’re discussing is also pretty impressive, a working lexer/parser

nwienert 4y ago

The answer is it’s very bad at large amounts of code. It has no skill at reasoning across a large amount of files where say you’d need to make many small changes in a handful of places and know that those changes all work together. These models are amazing at regurgitation. They have no skill at reasoning and any solution outside their training set. They have a distinct disability for precision in anything remotely novel or large. And they will bullshit subtle bugs all over the place at scale. Even a junior programmer would produce more value than the next generations of this, simply because they can take vague English instructions and actually make iterative progress across a huge codebase, test those changes for all sorts of vague constraints, and be precise. There’s no ability for that here.

passion__desire 4y ago
This tool would make for a great pair programmer and an intelligent interactive rubber duck
- birdyrooster 4y ago
  Except you must have the underlying knowledge to effectively review code. So how that would obviate a career in programming is a strange choice by the grandparent commenter. Anyone can rubber stamp bad code, but to stop bad code from entering projects you need experts.
  - aprdm 4y ago
    Most projects have plenty of bad code, nobody cares, as long as it solves the problem.
    - ad404b8a372f2b9 4y ago
      The bad code it produces is not just stylistically bad, it often doesn't solve the problem. It seems to be making the same kind of errors in code as we've seen in speech, you get a lot of plausible but subtly wrong hallucinated content that only someone who knows the subject can identify. It prefers probable output over matching the specifications so it'll add unnecessary branches just because it's seen them millions of times. Among errors I've seen it make there is bad indexing, adding conditions that serve no purpose or assuming a variable has a specific structure when it doesn't.

xur17 4y ago

> The question is how good it is to process larger chunks of code and makes changes to it. > People thinking about becoming programmers might need to rethink their plans if this one improves ... Very true, and this thought definitely crossed my mind as well. In the short term I imagine this is going to be like any other tool, it will increase your leverage, and make it possible for you to write code faster. For example, I fed it 100 lines of a Django model and asked it to write some query code for me. Produced something that was possible faster than I could have. The few slight deficiencies it did have (optimizations) basically felt like me doing a code review.

dopidopHN 4y ago

Hi, thanks for breaking it down. Do you write code close to the machine at $job? I wanna try with more feature and UI heavy request. I actually comment because of your edit in your response : you manually introduced bugs, reran the process, and it seen you broke it ( fine ) but was able to explain it to you in English ? Where is that comming from ? Adjacent comments and git comments?

makestuff 4y ago
I asked it to make some stuff in flutter and it worked well. I did notice sometimes it just adds a comment in the function saying “insert logic here for X” when it doesn’t know how to write it.
rightbyte 4y ago
> I actually comment because of your edit in your response : you manually introduced bugs, reran the process, and it seen you broke it ( fine ) but was able to explain it to you in English ? I fed it it's own sorting function but I added bugs to it and asked it to spot the bugs.

sage76 4y ago

I tried a hard level leetcode problem on it. Perfect results on the first attempt.

dvngnt_ 4y ago
it was trained with leetcode data though?
- sage76 4y ago
  Ah I didn't know that.

deleted 4y ago

[deleted]

nobody5050 4y ago

Something I’ve noticed in my own experimentations is that it’s really only limited by the length of its replies. It can happily generate perfect code but then stop in the middle of a function simply because the website only asked for 200 tokens or something. It’s really quite something to say that one of my gripes lies not in the model but in the user interface wrapping the model.

melagonster 4y ago

programmers are final worker. there are not no more jobs.

ChickeNES 4y ago

I already managed to get it to write an implementation of Pong and a Chip8 interpreter, both in C and using SDL2 for graphics/input/sound (it even wrote a function to generate the waveform for a beep for Chip8 and rebound sound for Pong!). I'm pretty sure I can get it to generate all the code for a Gameboy or NES emulator, I've taken a few tentative steps and it definitely can reproduce the memory map and generate code to parse iNES/GB cartridge headers