4 ms·
I've been using ChatGPT all weekend to generate code and what I found was this: * Its absurdly good at coding and following types. For example if you change
by hansonkd 4y ago
I've been using ChatGPT all weekend to generate code and what I found was this:
* Its absurdly good at coding and following types. For example if you change a type in Rust to be an Option, it will refactor the code to properly handle Options in the parts it used them. but it isn't perfect.
* It gets it well enough there. It can generate test cases so its easy to test to see if it works.
but in the end after hours and hours of trying to coax the AI, it was unable to do what I wanted, build a b-tree in Python. it built a binary tree just fine, but trying to have it generalize to a b-tree was a problem.
* it introduced many many many subtle errors. Like variables not being initialized or not splitting the children correctly.
* Its implementation worked when all keys inserted were in order, but not when they were out of order.
* it would miss and leave out variables.
* It would frequently have index errors from trying to access lists out of bounds.
* writing the code in Rust was almost impossible. it would constantly have wrong types or move errors.
Overall I couldn't recommend this to anyone without a strong CS background. It introduces far too many subtle bugs in code that are are almost impossible to review because the code it produces is so convincing that you go "hmm maybe it knows what it is talking about" but in the end you have no idea what you should trust.
Even the test cases that it generates can be deceptive. They look convincing but upon closer inspection, sometimes are just not really testing anything.
- lmarcos 4y agoSeems like it has the same coding abilities I had when I started to write code many years ago. Wonder how fast it can become better... I bet: exponentially faster than me.
- SxC97 4y agoI also tried to get it to generate some simple code examples. One was to generate a webpage with a button to add textboxes. Each textbox should have a unique remove button. When the site gets down to only one textbox on the page, it should not allow the user to remove the last textbox. After several iterations, it wasn't able to do it, often with hilarious results! (I asked it to only remove the _selected_ textbox, but if you clicked remove, it would just delete all the elements from the site) I think the real value for me would be using this to generate a starting point. I have a text document on my computer that has a long list of small coding projects that I want to get to at some point. But the activation energy required to stop watching Youtube and start coding is high enough that several projects have languished in my inbox for years. If I could just feed my ideas into chatGPT and get a starting point, it would be much easier to keep going and fix small errors/add additional features than to start a project from scratch.
- rakejake 4y agoCan you ask chatGPT why it wrote a certain line or why it made a certain decision? I still think explainability is missing in these models, but even if it able to come up with something, what is the guarantee that the explanation is not bullshit?
- qualudeheart 4y ago> Overall I couldn't recommend this to anyone without a strong CS background. It introduces far too many subtle bugs in code that are are almost impossible to review because the code it produces is so convincing that you go "hmm maybe it knows what it is talking about" but in the end you have no idea what you should trust. As someone using chatgpt all week— 30ish hours so far- and copilot since I got access early one. This fits with my xp very well. It’s like a junior programmer. You need the know how to know if your junior programmer is doing it right.