5 ms·

Building a back end using only OpenAI Codex

88 points by zegl 4y ago

I've published the sources for the code generation and the code that was generated on GitHub: https://github.com/sturdy-dev/codeball-todo-mvc https://github.com/sturdy-dev/codeball-todo-mvc I've been experimenting with merging prompts together, with a goal to write the full backend in a single prompt. On the form: > 1. Setup a flask web server > 2. Add a /add endpoint It works reasonably well, but it seems like it's loosing some precision in the prompts... The person that coined the term "prompt engineering" was right, it's really important to learn what words to use to get the AI to do exactly what you want it to do.

deleted 4y ago
[deleted]
kevincox 4y ago
This is mindblowing. The copilot autocomplete was very impressive but actually editing the existing code is incredible. I really want to see some examples of failed prompts and attempts to ask it to cache sqlite connections.
w1zzy 4y ago
How much time did it take to wrote app?
- zegl 4y ago
  It’s hard to say, since I was writing the blog post in parallel as I was making the app. But not too long, maybe an hour or two? I’m not a Python/Flask developer, so I guess that’s not too bad.
  - lxe 4y ago
    You can also use gpt3 to write most of the blog content. I'm willing to bet soon this is going to be everyone's workflow.
    - trention 4y ago
      Why would I want to read a blog post written by an AI about a program written by an AI besides novelty? The moment an input in a medium becomes primarily AI-generated is the moment I stop using said medium.
      - lxe 4y ago
        Because you would not be able to distinguish whether a paragraph is written by AI or not. It's not novelty, it's utility.
        trention 4y ago
        As I said, if I can't distinguish it, I will stop reading "it" at all. There is no utility in reading text that is not human-generated.
  - w1zzy 4y ago
    Thanks for info. I am gathering information about efficiency of using code generators. Btw. Nice work!
    - teslelover_12 4y ago
      yup its really cool
    - verdverm 4y ago
      I have a non-AI code generator if that would be of interest to your info gathering. It's definitely meant for developer efficiency over the entire software lifecycle. Happy to tell you all about it if you are doing interviews, calls, or the similar. https://github.com/hofstadter-io/hof https://github.com/hofstadter-io/hof | https://docs.hofstadter.io https://docs.hofstadter.io
xrd 4y ago
Why did you use sveltekit as the front end (as opposed to just svelte)? Typically SK is used when you want to have both front end and back end in the same app.
- zegl 4y ago
  It's mostly what I'm used to these days, codeball.ai is written in SK. I didn't end up using it, but SK also has a nice client side router!
daenz 4y ago
It's very cool, but from an auditing perspective, it's a nightmare. As a reviewer, I can't reason about the code in the same way that I could reason about human code, since there is no coherent formulation of the accomplished task. I can't say "why did it apply CORS to the entire flask app?" and expect reasoning that will fulfill my objective as a reviewer. So while it could help blast out large swaths of code quickly, it still needs an expert at the wheel to be accountable for the changes to reviewers.
- ShamelessC 4y ago
  As was the case before.
- nexuist 4y ago
  > I can't say "why did it apply CORS to the entire flask app?" and expect reasoning that will fulfill my objective as a reviewer. I'm not saying you're wrong, but can't you just ask the AI to include a comment explaining why it chose to apply CORS to the entire app? You can just keep asking it questions and maybe its reasoning would check out for most of them.
  - vosper 4y ago
    > just keep asking it questions and maybe its reasoning would check out for most of them. But the AI isn't reasoning... is it? Perhaps it could give an explanation, but you couldn't (currently) conflate that with any actual understanding of why it did what it did?
    - nexuist 4y ago
      It's not reasoning like a human, but if it's using code it memorized from the past it might be able to string along comments it memorized from the past into something that is relevant for the context it's being asked for.
sdwr 4y ago
I like how the captions scroll along to match the code samples, it's a fun reading experience. Got scared for a second (most of what I code is CRUD backends!), until I tried to see it from the perspective of a novice, where all of this is impenetrable anyway.
akkartik 4y ago
This needs a new format for source code. We could call it, oh, Literate Programming. Check in only the prompts to version control, expand them into code during CI, then file bugs when new releases of OpenAI cause regressions.
- thinkingkong 4y ago
  Like we just write gerkhin / cucumber all day?
- dwohnitmok 4y ago
  We cannot expect the code expansion to be deterministic with regards to the prompt without a severe reduction in AI capability. The utility of these prompts comes primarily from the fact that the AI is aware of a huge amount of context and can therefore infer what a prompt is "meant to do." If a prompter had to exhaustively specify the context it would be no different than coding in any normal programming language. That context necessarily changes over time. The same sentence 10 years ago might easily have a different contextual meaning than it does today.
  - verdverm 4y ago
    I think there is a middle ground. My work on a deterministic, ai-free code gen tool uses DSLs in a more abstract, declarative space. The details are handled in the templates and extra config from the input. Prisma, Atlas, and OpenAPI-generator are similar, with increasing complexity of input and DSL, respectively. I do like your point that context of natural language as input can change over time. I imagine it also would if trained on different source code, or even different target languages and technologies. I'm thinking that these AI could be simplified if their target was one of these middle ground abstractions in a DSL, letting fewer (expert) humans write the code via templates
nl 4y ago
The typo in the submission (like it's loosing [should be losing] some precision) is both inadvertently amusing (losing precision could well be described as being loose) and raises the question of how Codex would deal with missed typos in instructions.
- verdverm 4y ago
  My top-level comment is about a possible code typo that would appear much more serious https://news.ycombinator.com/item?id=32587425 https://news.ycombinator.com/item?id=32587425
verdverm 4y ago
The code on GitHub does not exactly match the post. In particular, the last section about adding seed data is shifted up a few lines on GitHub, into the first database call, making me wonder if it was a stored procedure or a bug. Did you have to correct output for the post? https://github.com/sturdy-dev/codeball-todo-mvc/blob/main/app.py https://github.com/sturdy-dev/codeball-todo-mvc/blob/main/ap... https://github.com/sturdy-dev/codeball-todo-mvc/commit/179923499913993e93a6fcf7a9f6e9c62e49cab0 https://github.com/sturdy-dev/codeball-todo-mvc/commit/17992...
- zegl 4y ago
  Nice find! I tried to be careful to make sure that everything aligned. The prompt used for the post seems to have been "before_first_request, before conn.close: if the tasks table is empty, add three rows" I'm updating the post and the sources!
  - verdverm 4y ago
    I think a video going through the series of prompts would be super interesting too
keyle 4y ago
Any instructions on how one goes ahead and play with openai codex themselves? Is this closed? Beta? or .. ? As a kid I used to dream to talk to the computer and it would make code happen as a repl. This appears to be close to it.
- sh4rks 4y ago
  If you're a student, copilot is free I believe
cbm-vic-20 4y ago
This is cool. Some thoughts: > beware: sometimes Codex writes code vulnerable to SQL injection. When that happens tough, I was able to prevent it by adding "safely" to the prompt. Oops. Anyway, is "the program" actually the prompts? Should that be committed into source control, so future you and others can figure out how the code was built? How long will it be until we can trust Codex enough that the Python code doesn't need to be committed? "Codex, create an Android CRUD UI for this OpenAPI document."
- smeagull 4y ago
  I wonder if the vulnerability could be detected at the embedding layer.
  - EGreg 4y ago
    Codex, create a dating site around the stable marriage algorithm. Kthxbai !
hirebackenddev 4y ago
This is something interesting. Thanks for posting such content.
citizenpaul 4y ago
> it's really important to learn what words to use to get the AI to do exactly what you want it to do. Now instead of spending years learning to do all that nasty troublesome coding you can just spend years learning to exactly phrase what you want the code generator to do. Wait.... is this an infinite loop joke?
- zegl 4y ago
  Kind of, sometimes coding with Codex is like having to debug/code review code from a developer that has just been through a 1-month "learn to code" bootcamp. Nothing against bootcampers, is's just that their output is non-intuitive...

88 points by zegl 4y ago

deleted 4y ago

[deleted]

kevincox 4y ago

This is mindblowing. The copilot autocomplete was very impressive but actually editing the existing code is incredible. I really want to see some examples of failed prompts and attempts to ask it to cache sqlite connections.

w1zzy 4y ago

How much time did it take to wrote app?

zegl 4y ago
It’s hard to say, since I was writing the blog post in parallel as I was making the app. But not too long, maybe an hour or two? I’m not a Python/Flask developer, so I guess that’s not too bad.
- lxe 4y ago
  You can also use gpt3 to write most of the blog content. I'm willing to bet soon this is going to be everyone's workflow.
  - trention 4y ago
    Why would I want to read a blog post written by an AI about a program written by an AI besides novelty? The moment an input in a medium becomes primarily AI-generated is the moment I stop using said medium.
    - lxe 4y ago
      Because you would not be able to distinguish whether a paragraph is written by AI or not. It's not novelty, it's utility.
      - trention 4y ago
        As I said, if I can't distinguish it, I will stop reading "it" at all. There is no utility in reading text that is not human-generated.
- w1zzy 4y ago
  Thanks for info. I am gathering information about efficiency of using code generators. Btw. Nice work!
  - teslelover_12 4y ago
    yup its really cool
  - verdverm 4y ago
    I have a non-AI code generator if that would be of interest to your info gathering. It's definitely meant for developer efficiency over the entire software lifecycle. Happy to tell you all about it if you are doing interviews, calls, or the similar. https://github.com/hofstadter-io/hof https://github.com/hofstadter-io/hof | https://docs.hofstadter.io https://docs.hofstadter.io

xrd 4y ago

Why did you use sveltekit as the front end (as opposed to just svelte)? Typically SK is used when you want to have both front end and back end in the same app.

zegl 4y ago
It's mostly what I'm used to these days, codeball.ai is written in SK. I didn't end up using it, but SK also has a nice client side router!

daenz 4y ago

It's very cool, but from an auditing perspective, it's a nightmare. As a reviewer, I can't reason about the code in the same way that I could reason about human code, since there is no coherent formulation of the accomplished task. I can't say "why did it apply CORS to the entire flask app?" and expect reasoning that will fulfill my objective as a reviewer. So while it could help blast out large swaths of code quickly, it still needs an expert at the wheel to be accountable for the changes to reviewers.

ShamelessC 4y ago
As was the case before.
nexuist 4y ago
> I can't say "why did it apply CORS to the entire flask app?" and expect reasoning that will fulfill my objective as a reviewer. I'm not saying you're wrong, but can't you just ask the AI to include a comment explaining why it chose to apply CORS to the entire app? You can just keep asking it questions and maybe its reasoning would check out for most of them.
- vosper 4y ago
  > just keep asking it questions and maybe its reasoning would check out for most of them. But the AI isn't reasoning... is it? Perhaps it could give an explanation, but you couldn't (currently) conflate that with any actual understanding of why it did what it did?
  - nexuist 4y ago
    It's not reasoning like a human, but if it's using code it memorized from the past it might be able to string along comments it memorized from the past into something that is relevant for the context it's being asked for.

sdwr 4y ago

I like how the captions scroll along to match the code samples, it's a fun reading experience. Got scared for a second (most of what I code is CRUD backends!), until I tried to see it from the perspective of a novice, where all of this is impenetrable anyway.

akkartik 4y ago

This needs a new format for source code. We could call it, oh, Literate Programming. Check in only the prompts to version control, expand them into code during CI, then file bugs when new releases of OpenAI cause regressions.

thinkingkong 4y ago
Like we just write gerkhin / cucumber all day?
dwohnitmok 4y ago
We cannot expect the code expansion to be deterministic with regards to the prompt without a severe reduction in AI capability. The utility of these prompts comes primarily from the fact that the AI is aware of a huge amount of context and can therefore infer what a prompt is "meant to do." If a prompter had to exhaustively specify the context it would be no different than coding in any normal programming language. That context necessarily changes over time. The same sentence 10 years ago might easily have a different contextual meaning than it does today.
- verdverm 4y ago
  I think there is a middle ground. My work on a deterministic, ai-free code gen tool uses DSLs in a more abstract, declarative space. The details are handled in the templates and extra config from the input. Prisma, Atlas, and OpenAPI-generator are similar, with increasing complexity of input and DSL, respectively. I do like your point that context of natural language as input can change over time. I imagine it also would if trained on different source code, or even different target languages and technologies. I'm thinking that these AI could be simplified if their target was one of these middle ground abstractions in a DSL, letting fewer (expert) humans write the code via templates

nl 4y ago

The typo in the submission (like it's loosing [should be losing] some precision) is both inadvertently amusing (losing precision could well be described as being loose) and raises the question of how Codex would deal with missed typos in instructions.

verdverm 4y ago
My top-level comment is about a possible code typo that would appear much more serious https://news.ycombinator.com/item?id=32587425 https://news.ycombinator.com/item?id=32587425

verdverm 4y ago

The code on GitHub does not exactly match the post. In particular, the last section about adding seed data is shifted up a few lines on GitHub, into the first database call, making me wonder if it was a stored procedure or a bug. Did you have to correct output for the post? https://github.com/sturdy-dev/codeball-todo-mvc/blob/main/app.py https://github.com/sturdy-dev/codeball-todo-mvc/blob/main/ap... https://github.com/sturdy-dev/codeball-todo-mvc/commit/179923499913993e93a6fcf7a9f6e9c62e49cab0 https://github.com/sturdy-dev/codeball-todo-mvc/commit/17992...

zegl 4y ago
Nice find! I tried to be careful to make sure that everything aligned. The prompt used for the post seems to have been "before_first_request, before conn.close: if the tasks table is empty, add three rows" I'm updating the post and the sources!
- verdverm 4y ago
  I think a video going through the series of prompts would be super interesting too

keyle 4y ago

Any instructions on how one goes ahead and play with openai codex themselves? Is this closed? Beta? or .. ? As a kid I used to dream to talk to the computer and it would make code happen as a repl. This appears to be close to it.

sh4rks 4y ago
If you're a student, copilot is free I believe

cbm-vic-20 4y ago

This is cool. Some thoughts: > beware: sometimes Codex writes code vulnerable to SQL injection. When that happens tough, I was able to prevent it by adding "safely" to the prompt. Oops. Anyway, is "the program" actually the prompts? Should that be committed into source control, so future you and others can figure out how the code was built? How long will it be until we can trust Codex enough that the Python code doesn't need to be committed? "Codex, create an Android CRUD UI for this OpenAPI document."

smeagull 4y ago
I wonder if the vulnerability could be detected at the embedding layer.
- EGreg 4y ago
  Codex, create a dating site around the stable marriage algorithm. Kthxbai !

hirebackenddev 4y ago

This is something interesting. Thanks for posting such content.

citizenpaul 4y ago

> it's really important to learn what words to use to get the AI to do exactly what you want it to do. Now instead of spending years learning to do all that nasty troublesome coding you can just spend years learning to exactly phrase what you want the code generator to do. Wait.... is this an infinite loop joke?

zegl 4y ago
Kind of, sometimes coding with Codex is like having to debug/code review code from a developer that has just been through a 1-month "learn to code" bootcamp. Nothing against bootcampers, is's just that their output is non-intuitive...