3 ms·

The output of LLMs is ... rarely well-designed. Well-documented (with often incorrect documentation), well-formatted for sure, but profoundly not well-designed,

The output of LLMs is ... rarely well-designed. Well-documented (with often incorrect documentation), well-formatted for sure, but profoundly not well-designed, unless you're asking for something so small that the design is trivial. Even with GPT-4, if you ask it for anything interesting, it often produces code that not only won't work, but that couldn't possibly work without a major rewrite. Not sure what you've been requesting if it's always been good output. Even when asking GPT-4 for docs I've had it hallucinate imaginary APIs and parameters more often than not. Maybe the questions I ask are not as common? Given my experiences, though, I wouldn't recommend it to anyone for fear it gave them profoundly bad advice.

nimithryn 3y ago
My experience as well. Heavy GPT-4 use (for a variety of things). Great for boilerplate, great for retrieving well-known examples from documentation, saves a fair amount of time typing and googling, but often completely wrong (majorly and subtly) and anything non-trivial I have to do myself. Great tool! Saves a ton of time! Not a dev replacement (yet)
- MeImCounting 3y ago
  Now I am definitely doing much simpler things I would wager than you folks are but I have found that with a bit of back and forth you can get pretty good results that work with only a bit of revision. I have found reminding it of the purpose or goals of whatever it is youre working on at the moment tends to make the output a bit more consistent
  - koreth1 3y ago
    > with a bit of back and forth you can get pretty good results that work with only a bit of revision The problem for me is that the "back and forth" and "a bit of revision" steps very often end up taking more time than writing the code myself would have.
    - MeImCounting 3y ago
      Thats because you actually know what youre doing haha. In all seriousness I am not a software engineer and GPT has enabled me to build things in a couple weeks that would have taken me months of effort to create otherwise. I am sure an actual software engineer could have made those same tools in a day or two but its still incredible for my use case.
    - nimithryn 3y ago
      Yeah, also, you tend to need to know the answer before you prompt it. It's kind of like a Ouija board, in that you get out what you expect.
dkersten 3y ago
I’ve come to the conclusion that GPT produces code at a level of a new graduate at best. In actually getting it to solve something more or less on its own, it did ok on simple tasks and failed as soon as requirements became a bit more nuanced or specific. It’s also not very good at thinking out of the box, it’s solutions are all very clearly tied to its training data, meaning it struggled doing anything that strayed too far into the abstract or different. However it’s been great at being my rubber duck and it’s been great as a tool for helping me eg write complex SQL queries — never without me being a key part of the loop, but as a tool to help me fill in gaps in my own skills or understanding. That is, it amplified my abilities. It was also pretty good at creating interesting metaphors for existing concepts, explaining terminology and even explaining bits of code I gave it.

nimithryn 3y ago

My experience as well. Heavy GPT-4 use (for a variety of things). Great for boilerplate, great for retrieving well-known examples from documentation, saves a fair amount of time typing and googling, but often completely wrong (majorly and subtly) and anything non-trivial I have to do myself. Great tool! Saves a ton of time! Not a dev replacement (yet)

MeImCounting 3y ago
Now I am definitely doing much simpler things I would wager than you folks are but I have found that with a bit of back and forth you can get pretty good results that work with only a bit of revision. I have found reminding it of the purpose or goals of whatever it is youre working on at the moment tends to make the output a bit more consistent
- koreth1 3y ago
  > with a bit of back and forth you can get pretty good results that work with only a bit of revision The problem for me is that the "back and forth" and "a bit of revision" steps very often end up taking more time than writing the code myself would have.
  - MeImCounting 3y ago
    Thats because you actually know what youre doing haha. In all seriousness I am not a software engineer and GPT has enabled me to build things in a couple weeks that would have taken me months of effort to create otherwise. I am sure an actual software engineer could have made those same tools in a day or two but its still incredible for my use case.
  - nimithryn 3y ago
    Yeah, also, you tend to need to know the answer before you prompt it. It's kind of like a Ouija board, in that you get out what you expect.

dkersten 3y ago

I’ve come to the conclusion that GPT produces code at a level of a new graduate at best. In actually getting it to solve something more or less on its own, it did ok on simple tasks and failed as soon as requirements became a bit more nuanced or specific. It’s also not very good at thinking out of the box, it’s solutions are all very clearly tied to its training data, meaning it struggled doing anything that strayed too far into the abstract or different. However it’s been great at being my rubber duck and it’s been great as a tool for helping me eg write complex SQL queries — never without me being a key part of the loop, but as a tool to help me fill in gaps in my own skills or understanding. That is, it amplified my abilities. It was also pretty good at creating interesting metaphors for existing concepts, explaining terminology and even explaining bits of code I gave it.