6 ms·

> I recently asked ChatGPT to produce similar CSS and it did so flawlessly. I use ChatGPT every day for many tasks in my work and find it very helpful, but I s

> I recently asked ChatGPT to produce similar CSS and it did so flawlessly. I use ChatGPT every day for many tasks in my work and find it very helpful, but I simply do not believe this. > The article points this out: middling generalists can now compete with specialists. I'd say it might allow novices to compete with middling generalists, but even that is a stretch. On the contrary, ChatGPT is actually best suited to use by a specialist who has enough contextual knowledge to construct targeted prompts & can then verify & edit the responses into something optimal.

tlarkworthy 3y ago
I wrote just the tool to optimize AI in the hand of a coding expert https://observablehq.com/@tomlarkworthy/robocoop https://observablehq.com/@tomlarkworthy/robocoop
jorams 3y ago
I agree with this. There are cases where it produces good results, but there are also cases where it produces bs, and it's not always obvious. I find it to work fine for cases where I know what I want but could use a starting point, but it often invents or misunderstands all kinds of things. The most frustrating situations are those where it invents a function that would miraculously do what's necessary, I tell it that function does not exist, it apologizes, shuffles the code around a bit and invents a different function, etc. It's the most annoying kind of debugging there is.
- lucideer 3y ago
  Another very obvious thing it does when it comes to code is take the most common misconceptions & anti-patterns used within the programming community & repeats them in an environment where there's no-one to comment. People have critiqued Stack Overflow for having so many "wrong" answers with green checkmarks, but at least those threads have surrounding context & discussion. A case in point: I asked ChatGPT to give me some code for password complexity validation. It gave me perfectly working code that took a password and validated it against X metrics. Obviously the metrics are garbage, but the code works, and what inexperienced developer would be any the wiser. The only way to get ChatGPT to generate something "correct" there would be to tell it algorithmically what you want (e.g. "give me a function measuring information entropy of inputs", etc.) - you could ask it 50 times for a password validator: every one may execute successfully & produce a desired UI output for a web designer, but be effectively nonsense.
- katzgrau 3y ago
  > there are also cases where it produces bs, and it's not always obvious Particularly annoying because I wind up risking not actually saving time because it’s producing subtle bugs that I wouldn’t have written myself. So, you save yourself the time of thought and research at the risk of going down new and mysterious rabbit holes
  - redblacktree 3y ago
    For me the trick to avoiding this trap is to limit usage to small areas of code, test frequently, and know its limits. I love using copilot/GPT for boilerplate stuff.
- nkozyra 3y ago
  > There are cases where it produces good results, but there are also cases where it produces bs, and it's not always obvious. Pessimistically, this is the medium term role I see for a lot of devs. Less actual development, more assembly of pieces and being good enough at cleaning up generated code. If an LLM can get you even 25% there most of the time, that's a massive disruption of this industry.
  - zrobotics 3y ago
    I mean, especially in webdev we've been heading in that direction for a while now anyway. So much of the job is already just wiring up different npm packages and APIs that someone else has written. I've read substantially similar comments back in the mid 2010s about how people weren't learning the fundamentals and just pulling things like left pad off of a repo. That did cause a disruption in how people coded by abstraction away many of the problems and making the job more about integrating different things together.
steveBK123 3y ago
That's about my experience. The worst dev on my team uses ChatGPT a lot, and its facilitated him producing more bad code more quickly. I'm not sure it's a win for anyone, and he's still unlikely to be with the team in a year. It allows a dev who doesn't care about their craft or improving to generate code without learning anything. The code they generate today or a year from today is the same quality. Part of it is that it allows devs who lean into overcomplicating things to do so even more. The solutions are never a refinement of what already exists, but patch on top of patch on top of patch of complexity. ChatGPT is not going to tell you how to design a system, architect properly, automate, package, test, deploy, etc. For the team it means there's a larger mess of a code base to rewrite.
- happycube 3y ago
  On the flip side, one can use ChatGPT as only a starting point and to learn from there. One isn't stuck with actually using what it outputs verbtim, and really shouldn't until at least a hypothetical GPT-6 or 7... and to use it fully now, one has to know how to nudge it when it goes into a bad direction. So overall it's more an amplifier than anything else.
  - lucideer 3y ago
    > So overall it's more an amplifier than anything else. Overall it would be an amplifier if that were how the majority used it. Sadly I don't believe that to be the case.
    - aswanson 3y ago
      That's been the case with every technology made by man since fire.
      - lucideer 3y ago
        Yup. We should embrace it, but without be naïve about what great things it's bringing us :)
dijit 3y ago
> ChatGPT is actually best suited to use by a specialist who has enough contextual knowledge to construct targeted prompts This is my take also. ChatGPT for novices is dangerous, its the equivalent of a calculator. If you don't know your expected output you're just wrong faster. But if you know what to expect, whats your bounds and how to do it normally anyway, it can make you faster.
schnable 3y ago
> ChatGPT is actually best suited to use by a specialist who has enough contextual knowledge to construct targeted prompts & can then verify & edit the responses into something optimal. I agree with this, but what that means is that specialists will be able to create next generation tools--across all professions including coding--that do supercharge novices and generalists to do more.

tlarkworthy 3y ago

I wrote just the tool to optimize AI in the hand of a coding expert https://observablehq.com/@tomlarkworthy/robocoop https://observablehq.com/@tomlarkworthy/robocoop

jorams 3y ago

I agree with this. There are cases where it produces good results, but there are also cases where it produces bs, and it's not always obvious. I find it to work fine for cases where I know what I want but could use a starting point, but it often invents or misunderstands all kinds of things. The most frustrating situations are those where it invents a function that would miraculously do what's necessary, I tell it that function does not exist, it apologizes, shuffles the code around a bit and invents a different function, etc. It's the most annoying kind of debugging there is.

lucideer 3y ago
Another very obvious thing it does when it comes to code is take the most common misconceptions & anti-patterns used within the programming community & repeats them in an environment where there's no-one to comment. People have critiqued Stack Overflow for having so many "wrong" answers with green checkmarks, but at least those threads have surrounding context & discussion. A case in point: I asked ChatGPT to give me some code for password complexity validation. It gave me perfectly working code that took a password and validated it against X metrics. Obviously the metrics are garbage, but the code works, and what inexperienced developer would be any the wiser. The only way to get ChatGPT to generate something "correct" there would be to tell it algorithmically what you want (e.g. "give me a function measuring information entropy of inputs", etc.) - you could ask it 50 times for a password validator: every one may execute successfully & produce a desired UI output for a web designer, but be effectively nonsense.
katzgrau 3y ago
> there are also cases where it produces bs, and it's not always obvious Particularly annoying because I wind up risking not actually saving time because it’s producing subtle bugs that I wouldn’t have written myself. So, you save yourself the time of thought and research at the risk of going down new and mysterious rabbit holes
- redblacktree 3y ago
  For me the trick to avoiding this trap is to limit usage to small areas of code, test frequently, and know its limits. I love using copilot/GPT for boilerplate stuff.
nkozyra 3y ago
> There are cases where it produces good results, but there are also cases where it produces bs, and it's not always obvious. Pessimistically, this is the medium term role I see for a lot of devs. Less actual development, more assembly of pieces and being good enough at cleaning up generated code. If an LLM can get you even 25% there most of the time, that's a massive disruption of this industry.
- zrobotics 3y ago
  I mean, especially in webdev we've been heading in that direction for a while now anyway. So much of the job is already just wiring up different npm packages and APIs that someone else has written. I've read substantially similar comments back in the mid 2010s about how people weren't learning the fundamentals and just pulling things like left pad off of a repo. That did cause a disruption in how people coded by abstraction away many of the problems and making the job more about integrating different things together.

steveBK123 3y ago

That's about my experience. The worst dev on my team uses ChatGPT a lot, and its facilitated him producing more bad code more quickly. I'm not sure it's a win for anyone, and he's still unlikely to be with the team in a year. It allows a dev who doesn't care about their craft or improving to generate code without learning anything. The code they generate today or a year from today is the same quality. Part of it is that it allows devs who lean into overcomplicating things to do so even more. The solutions are never a refinement of what already exists, but patch on top of patch on top of patch of complexity. ChatGPT is not going to tell you how to design a system, architect properly, automate, package, test, deploy, etc. For the team it means there's a larger mess of a code base to rewrite.

happycube 3y ago
On the flip side, one can use ChatGPT as only a starting point and to learn from there. One isn't stuck with actually using what it outputs verbtim, and really shouldn't until at least a hypothetical GPT-6 or 7... and to use it fully now, one has to know how to nudge it when it goes into a bad direction. So overall it's more an amplifier than anything else.
- lucideer 3y ago
  > So overall it's more an amplifier than anything else. Overall it would be an amplifier if that were how the majority used it. Sadly I don't believe that to be the case.
  - aswanson 3y ago
    That's been the case with every technology made by man since fire.
    - lucideer 3y ago
      Yup. We should embrace it, but without be naïve about what great things it's bringing us :)

dijit 3y ago

> ChatGPT is actually best suited to use by a specialist who has enough contextual knowledge to construct targeted prompts This is my take also. ChatGPT for novices is dangerous, its the equivalent of a calculator. If you don't know your expected output you're just wrong faster. But if you know what to expect, whats your bounds and how to do it normally anyway, it can make you faster.

schnable 3y ago

> ChatGPT is actually best suited to use by a specialist who has enough contextual knowledge to construct targeted prompts & can then verify & edit the responses into something optimal. I agree with this, but what that means is that specialists will be able to create next generation tools--across all professions including coding--that do supercharge novices and generalists to do more.