4 ms·

This is about what I expected if my own experimentation extrapolated. The code quality was too inconsistent to replace human labor for many uses. However, it w

This is about what I expected if my own experimentation extrapolated. The code quality was too inconsistent to replace human labor for many uses. However, it was good for automating boilerplate and for one-off utilities that were tedious to write. In those two categories, many new uses are similar to previous uses in the training data. So, extrapolation is easier for such jobs. The abstract suggests that’s the kind of use they’re doing. That’s also why they’re not usually fixing the bugs. We can often ignore or work around bugs in such use cases.

devbent 2y ago
My newest open source project[1] is ~90% written by LLM, but I work in the field so there is a chance my ability to manipulate LLMs is above average. It is also a tiny project, just a single microservice, although my upcoming open source project is fairly substantial and I suspect it'll be at least 30% LLM generated. The fact is LLMs are really good at generating chunks of code, sometimes even entire files, but they aren't that good at making systematic changes to large swaths of existing code. They can be made to work for that scenario with tons of prompt engineering and prompt chaining, but that isn't where LLMs shine. For example almost all of the HTML in that project was made by Claude. I then modified the code and re-uploaded it and asked Claude to add collapsible info boxes to parts of the page after the fact, and Claude completely failed at the task. Still though I am seeing an incredible increase in my development velocity. [1] https://www.github.com/devlinb/escaperoom https://www.github.com/devlinb/escaperoom -ugly UI, designed to help people create escape rooms for LLMs. A hosted version will be up later today!
PaulHoule 2y ago
I find Copilot is often great for writing a single line of code that would otherwise take some research to write. I also had it to write a DOSKEY macro for CMD.EXE despite not doing serious scripting w/ CMD.EXE since 1989 or so. It confused the / and \ but I might have done the same and when I asked it about the unfamiliar constructs I saw it explained them well.

devbent 2y ago

My newest open source project[1] is ~90% written by LLM, but I work in the field so there is a chance my ability to manipulate LLMs is above average. It is also a tiny project, just a single microservice, although my upcoming open source project is fairly substantial and I suspect it'll be at least 30% LLM generated. The fact is LLMs are really good at generating chunks of code, sometimes even entire files, but they aren't that good at making systematic changes to large swaths of existing code. They can be made to work for that scenario with tons of prompt engineering and prompt chaining, but that isn't where LLMs shine. For example almost all of the HTML in that project was made by Claude. I then modified the code and re-uploaded it and asked Claude to add collapsible info boxes to parts of the page after the fact, and Claude completely failed at the task. Still though I am seeing an incredible increase in my development velocity. [1] https://www.github.com/devlinb/escaperoom https://www.github.com/devlinb/escaperoom -ugly UI, designed to help people create escape rooms for LLMs. A hosted version will be up later today!

PaulHoule 2y ago

I find Copilot is often great for writing a single line of code that would otherwise take some research to write. I also had it to write a DOSKEY macro for CMD.EXE despite not doing serious scripting w/ CMD.EXE since 1989 or so. It confused the / and \ but I might have done the same and when I asked it about the unfamiliar constructs I saw it explained them well.