6 ms·

It managed to write, in one shot, a working λ-calculus parser, using a very specific programming style I asked it in JavaScript, and then translated it all to P

It managed to write, in one shot, a working λ-calculus parser, using a very specific programming style I asked it in JavaScript, and then translated it all to Python, including sarcastic, rhyming GLaDOS comments. https://twitter.com/VictorTaelin/status/1635726202231988225 https://twitter.com/VictorTaelin/status/1635726202231988225 It also seems to be extremely competent at writing Agda types and proofs. We need some tool that highly integrates it with entire codebases, allowing for major refactors to be requested as a prompt. That would be groundbreaking (understatement of the year), specially if coupled with a strongly typed language that prevents it from accidentally breaking things up.

sualehasif 4y ago
Yeah, its really cool how good it is. I personally am addicted to it.
deleted 4y ago
[deleted]
amelius 4y ago
I don't think this is a good test. Every CS graduate student has to write a lambda-calculus parser. There must be thousands of implementations on the web. It really is not strange that GPT-4 can reproduce this.
- capableweb 4y ago
  Let's invent a better test then. What could be an example task that isn't widely available on the internet already? I'm having a hard time coming up with anything that isn't reproduced in many places.
  - palijer 4y ago
    Take something from advent of code and change it up a bit seems like a good test
    - Jtsummers 4y ago
      Give it Synacor Challenge, just the spec, and see if it can pull it off. Fewer of those solutions out there. It just went offline recently, but Aneurysm9 has preserved the problem spec, their binary, and the checksum of the codes for their binary to check against. https://github.com/Aneurysm9/vm_challenge https://github.com/Aneurysm9/vm_challenge The program will, likely, need to be amended to get the last code (last 2 codes? been a while) so you can see how it would handle updating for the new requirements.
    - rootusrootus 4y ago
      Ha, I gave Bing Chat something from Advent of Code and it correctly identified that it came from Advent of Code (without that being anywhere in the prompt). It provided a solution, but given that it identified the source of the question I don't think it was a good test. As you say, maybe changing some values will help.
- deleted 4y ago
  [deleted]
- deleted 4y ago
  [deleted]
- LightMachine 4y ago
  I sort of agree, but it still amazes me that it can get the code is correct, even though I asked for a highly specific style (i.e., use of recursion, representing "None" as null, the format of the JSON, making local functions, etc.). So, even thought it has never seen that exact implementation, it still assembles a working function that just works. If it was just mixing up different code it recalled from memory, it would likely have a bunch of silly errors here and there that I'd have to fix manually, but no, it just works. That's what impressed me.
- worrycue 4y ago
  Frankly, if ChatGPT is only good at reproducing commonly written code, I don’t think it will impact the profession much given that code reuse is already a thing and lots are distributed free on the internet.
  - LightMachine 4y ago
    Even if that was the case, it'd still be massively useful because it would allow code to be transposed between languages idiomatically. For example, certain things like game engines were only implemented in very messy C++ code. If GPT understands how these libraries work, could it recreate all these game engines as cleaned up, elegant Haskell code?
    - famouswaffles 4y ago
      Bilingual LLMs are already excellent at translating human languages. I imagine we'll find that GPT can translate between existing programming languages very well. We could soon see a future where "Damn this useful paper/code is in x language. Have to wait for someone to port the code over to y language" is a thing of the past.
      - dwohnitmok 4y ago
        They are excellent. Not quite human level, but very very close. I was curious about Chinese-English translation capabilities of the latest crop of models and on more difficult texts a bilingual model like GLM-130B makes several errors per page while GPT-4 is down to probably just around one. Interested to see how that plays out for programming languages.
  - CapstanRoller 4y ago
    Most code produced is commonly-written code. A sizable portion of software devs ultimately work on something that, at its core, follows the basic CRUD pattern. The day-to-day stuff also involves a lot of boilerplate -- "public static void main" has paid a lot of mortgages over the years.
jhp123 4y ago
it looks like the lam parser should throw, as is it will consume input then fall through to app. whitespace handling is also not correct. I don't believe I've seen one correct program written by GPT yet. edit: try parsing "λ(x x)" for example, or "λ x. x"

sualehasif 4y ago

Yeah, its really cool how good it is. I personally am addicted to it.

deleted 4y ago

[deleted]

amelius 4y ago

I don't think this is a good test. Every CS graduate student has to write a lambda-calculus parser. There must be thousands of implementations on the web. It really is not strange that GPT-4 can reproduce this.

capableweb 4y ago
Let's invent a better test then. What could be an example task that isn't widely available on the internet already? I'm having a hard time coming up with anything that isn't reproduced in many places.
- palijer 4y ago
  Take something from advent of code and change it up a bit seems like a good test
  - Jtsummers 4y ago
    Give it Synacor Challenge, just the spec, and see if it can pull it off. Fewer of those solutions out there. It just went offline recently, but Aneurysm9 has preserved the problem spec, their binary, and the checksum of the codes for their binary to check against. https://github.com/Aneurysm9/vm_challenge https://github.com/Aneurysm9/vm_challenge The program will, likely, need to be amended to get the last code (last 2 codes? been a while) so you can see how it would handle updating for the new requirements.
  - rootusrootus 4y ago
    Ha, I gave Bing Chat something from Advent of Code and it correctly identified that it came from Advent of Code (without that being anywhere in the prompt). It provided a solution, but given that it identified the source of the question I don't think it was a good test. As you say, maybe changing some values will help.
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
LightMachine 4y ago
I sort of agree, but it still amazes me that it can get the code is correct, even though I asked for a highly specific style (i.e., use of recursion, representing "None" as null, the format of the JSON, making local functions, etc.). So, even thought it has never seen that exact implementation, it still assembles a working function that just works. If it was just mixing up different code it recalled from memory, it would likely have a bunch of silly errors here and there that I'd have to fix manually, but no, it just works. That's what impressed me.
worrycue 4y ago
Frankly, if ChatGPT is only good at reproducing commonly written code, I don’t think it will impact the profession much given that code reuse is already a thing and lots are distributed free on the internet.
- LightMachine 4y ago
  Even if that was the case, it'd still be massively useful because it would allow code to be transposed between languages idiomatically. For example, certain things like game engines were only implemented in very messy C++ code. If GPT understands how these libraries work, could it recreate all these game engines as cleaned up, elegant Haskell code?
  - famouswaffles 4y ago
    Bilingual LLMs are already excellent at translating human languages. I imagine we'll find that GPT can translate between existing programming languages very well. We could soon see a future where "Damn this useful paper/code is in x language. Have to wait for someone to port the code over to y language" is a thing of the past.
    - dwohnitmok 4y ago
      They are excellent. Not quite human level, but very very close. I was curious about Chinese-English translation capabilities of the latest crop of models and on more difficult texts a bilingual model like GLM-130B makes several errors per page while GPT-4 is down to probably just around one. Interested to see how that plays out for programming languages.
- CapstanRoller 4y ago
  Most code produced is commonly-written code. A sizable portion of software devs ultimately work on something that, at its core, follows the basic CRUD pattern. The day-to-day stuff also involves a lot of boilerplate -- "public static void main" has paid a lot of mortgages over the years.

jhp123 4y ago

it looks like the lam parser should throw, as is it will consume input then fall through to app. whitespace handling is also not correct. I don't believe I've seen one correct program written by GPT yet. edit: try parsing "λ(x x)" for example, or "λ x. x"