3 ms·

Conversely, I routinely use LLMs to do development tasks now. My last bit of greenfield work, I had an LLM stub out an entire API for it. When I needed to marsh

Conversely, I routinely use LLMs to do development tasks now. My last bit of greenfield work, I had an LLM stub out an entire API for it. When I needed to marshal a fairly complex JSON object into a well-defined object in my code, including several validations, due to some security controls, the LLM taught me about a library I wasn't familiar with in this language. Or, a few days ago, I had the need to run the same CLI commands a few dozen times with slightly differing parameters. Unfortunately, the CLI only exist in Windows, and needed to be called on multiple hosts in a Windows environment. I could probably do this on Linux using bash and/or Python, but in Windows, it was way easier to just have the LLM write a PowerShell script for me. LLMs aren't the best at dealing with proprietary codebases (yet), but I'm sure that will come. In the meantime, they're really useful for abstracting away mundane work in a way that's much more user-friendly than your IDE probably offers, and they often help me spot issues with my assumptions, as well.

nunez 3y ago
But what if you enjoy doing those things by hand?
- jwestbury 3y ago
  Nothing wrong with that. I don't, though -- so LLMs have made me more productive with less annoyance. (My personal struggle is figuring out when to stop trying to use the hammer that is LLMs. I've definitely fallen victim to the sunk cost fallacy here.)
eichin 3y ago
hmm, is the powershell example something you don't expect to have to do again, so it's not worth really understanding the details? (and did you feel you needed to verify the output, or was just running it enough? Not trying to judge here! Just curious because nngroup just published a user study that showed only a tiny percentage of people actually cross-checking LLM-assistant output - not that it was wrong, they weren't checking either, just that most of the users in their study didn't feel it was necessary.)
- jwestbury 3y ago
  It's something which probably needs to be done again -- I included a few parameters in the script that allow it to be used for a limited set of similar workflows in the future. I read over the script, didn't spend loads of time understanding it, but the logic looked roughly correct. I generally read the code the LLM generates, then test it. I also try to include dry run modes whenever possible, so I can validate the mutating behaviour is correct before actually running any mutating commands -- I'm far more likely to do this with LLM-generated scripts than my own code, I've found, though perhaps that's just laziness. :)
- FranksTV 3y ago
  There are a ton of development tasks that I do infrequently enough that I have to google how to do them every time, because I forget how to do them. For those things, I can just ask Chat-GPT to write the first draft of it, and it saves me about 80% of the time. I always have to end up doing a few edits, but it works out. Also dropping in an indecipherable page of logs and immediately getting the source of an error with at least a suggestion of a direction is really useful.

nunez 3y ago

But what if you enjoy doing those things by hand?

jwestbury 3y ago
Nothing wrong with that. I don't, though -- so LLMs have made me more productive with less annoyance. (My personal struggle is figuring out when to stop trying to use the hammer that is LLMs. I've definitely fallen victim to the sunk cost fallacy here.)

eichin 3y ago

hmm, is the powershell example something you don't expect to have to do again, so it's not worth really understanding the details? (and did you feel you needed to verify the output, or was just running it enough? Not trying to judge here! Just curious because nngroup just published a user study that showed only a tiny percentage of people actually cross-checking LLM-assistant output - not that it was wrong, they weren't checking either, just that most of the users in their study didn't feel it was necessary.)

jwestbury 3y ago
It's something which probably needs to be done again -- I included a few parameters in the script that allow it to be used for a limited set of similar workflows in the future. I read over the script, didn't spend loads of time understanding it, but the logic looked roughly correct. I generally read the code the LLM generates, then test it. I also try to include dry run modes whenever possible, so I can validate the mutating behaviour is correct before actually running any mutating commands -- I'm far more likely to do this with LLM-generated scripts than my own code, I've found, though perhaps that's just laziness. :)
FranksTV 3y ago
There are a ton of development tasks that I do infrequently enough that I have to google how to do them every time, because I forget how to do them. For those things, I can just ask Chat-GPT to write the first draft of it, and it saves me about 80% of the time. I always have to end up doing a few edits, but it works out. Also dropping in an indecipherable page of logs and immediately getting the source of an error with at least a suggestion of a direction is really useful.