4 ms·

If you ask it to start explaining/extending/implementing classes/interfaces, start asking it to reason about leaky abstractions or when it’s appropriate to use

If you ask it to start explaining/extending/implementing classes/interfaces, start asking it to reason about leaky abstractions or when it’s appropriate to use Mixins, it frequently generates code that either doesn’t work or just misses the point. It’s especially hard because some of these problems actually are not tractable in the language you’re taking about due to their features, or due to limitations of your code/your knowledge it may suggest you do something slightly worse because you haven’t prompted it on factories or other design patterns that you might want to use.

simonw 3y ago
There are an almost unlimited number of ways you can prompt it that will produce obviously bad results. The trick to using it effectively is to figure out the ways of prompting it that produce GOOD results.
- zerbinxx 3y ago
  Right, but obviously you can see the unknown unknown problem or Dunning-Krueger popping up when software developers who don’t know everything (unlike me, of course) start using it extensively without knowing the ways in which their prompt engineering is flawed.
  - simonw 3y ago
    One of the big open questions in LLM-assisted learning is whether it helps or hinders less experienced developers. At this point I'm pretty confident it helps them. They need to understand that it's not infallible, but I think most people figure that out pretty quickly after using it for more than a few days. There's absolutely a skill to using these things well, and it's a difficult one to teach because it's based more on intuition that you develop over time. The key thing is helping people understand the strengths and limitations of these tools. That's something I do think we can teach.
- lawlessone 3y ago
  >The trick to using it effectively is to figure out the ways of prompting it that produce GOOD results Might as well just learn to code then.
  - simonw 3y ago
    My argument is that AI-assistance makes it easier and faster to learn to code.
- jazzyjackson 3y ago
  it's not just good prompts vs bad prompts, there's luck involved, since the result is determined by dice rolls. I used to be smug about getting good results from GPT while coworkers failed to get it to produce anything useful, but over time my luck changed, and GPT stopped nailing it. If you remember there was a bunch of conspiracy talk that GPT got lobotomized, was no longer writing full programs anymore etc, but what really happened is people who got used to rolling 7s started rolling 2s and 3s and shook their fists in anger at their change of luck.
J_Shelby_J 3y ago
I’ve had this exact experience with this topic learning python and designing my first app. GPT frequently sent my down the wrong path and wasted my time when I could of just used it to learn about a topic and make my own decisions. It does not hand recursive thinking well at all.
- zerbinxx 3y ago
  Agreed. A good example of this (in Python) is to explore solutions to problems that use decorators/a more functional style vs. solutions that use classes and a more OO style. Usually both are totally good. The code GPT generates can be totally functional, yet inappropriate in context. Another good one is asking it to translate code from JS to a language without real “interfaces” and see if it uses a base class pattern or not. I review junior devs and lecture them extensively about stuff like this. (Yes, I am fun at parties.)
Terr_ 3y ago
Also, as you go down that path, you may hit it's limits on tokens/memory, and it forgets what you were originally talking about.

simonw 3y ago

There are an almost unlimited number of ways you can prompt it that will produce obviously bad results. The trick to using it effectively is to figure out the ways of prompting it that produce GOOD results.

zerbinxx 3y ago
Right, but obviously you can see the unknown unknown problem or Dunning-Krueger popping up when software developers who don’t know everything (unlike me, of course) start using it extensively without knowing the ways in which their prompt engineering is flawed.
- simonw 3y ago
  One of the big open questions in LLM-assisted learning is whether it helps or hinders less experienced developers. At this point I'm pretty confident it helps them. They need to understand that it's not infallible, but I think most people figure that out pretty quickly after using it for more than a few days. There's absolutely a skill to using these things well, and it's a difficult one to teach because it's based more on intuition that you develop over time. The key thing is helping people understand the strengths and limitations of these tools. That's something I do think we can teach.
lawlessone 3y ago
>The trick to using it effectively is to figure out the ways of prompting it that produce GOOD results Might as well just learn to code then.
- simonw 3y ago
  My argument is that AI-assistance makes it easier and faster to learn to code.
jazzyjackson 3y ago
it's not just good prompts vs bad prompts, there's luck involved, since the result is determined by dice rolls. I used to be smug about getting good results from GPT while coworkers failed to get it to produce anything useful, but over time my luck changed, and GPT stopped nailing it. If you remember there was a bunch of conspiracy talk that GPT got lobotomized, was no longer writing full programs anymore etc, but what really happened is people who got used to rolling 7s started rolling 2s and 3s and shook their fists in anger at their change of luck.

J_Shelby_J 3y ago

I’ve had this exact experience with this topic learning python and designing my first app. GPT frequently sent my down the wrong path and wasted my time when I could of just used it to learn about a topic and make my own decisions. It does not hand recursive thinking well at all.

zerbinxx 3y ago
Agreed. A good example of this (in Python) is to explore solutions to problems that use decorators/a more functional style vs. solutions that use classes and a more OO style. Usually both are totally good. The code GPT generates can be totally functional, yet inappropriate in context. Another good one is asking it to translate code from JS to a language without real “interfaces” and see if it uses a base class pattern or not. I review junior devs and lecture them extensively about stuff like this. (Yes, I am fun at parties.)

Terr_ 3y ago

Also, as you go down that path, you may hit it's limits on tokens/memory, and it forgets what you were originally talking about.