5 ms·

At least in my experience, it excels in blank canvas projects. Where you've got nothing and want something pretty basic. The tools can probably set up a fresh R

At least in my experience, it excels in blank canvas projects. Where you've got nothing and want something pretty basic. The tools can probably set up a fresh React project faster than me. But at least every time I've tried them on an actual work repo they get reduced to almost useless. Which is why they generate so much hype. They are perfect for tech demos, then management wonders why they aren't seeing results in the real world.

herpdyderp 1y ago
I've had great success with GPT5 in existing projects because its agent mode is very good (the best I've seen so far) at analyzing the existing codebase and then writing code that feels like it fits in already (without prompt engineering on my part). I still agree that AI is particularly good on fresh projects though.
- SchemaLoad 1y ago
  Could be that there is a huge difference in the products. Last few companies have given me Github Copilot which I find entirely useless, I found the automatic suggestions more distracting than useful, and the fix and explain functions never work. But maybe if you burn $1000/day on Claude Code it works a lot better. And then companies see the results from that and wonder why they aren't getting it spending a couple of dollars on Copilot.
  - herpdyderp 1y ago
    I use GitHub Copilot from work in agent mode with GPT5 and it’s great! I don’t use the suggestions, fix, or explain features, I agree they’re almost always not helpful.
    - SchemaLoad 1y ago
      Is that a separate product you have to pay for? I've never seen an agent feature in the VS code plugin.
      - adithyassekhar 1y ago
        It's there next to the model selector in chat.
tomrod 1y ago
Exactly. It quickly builds a lot of technical debt that must be paid down, especially for people writing code in areas they aren't deep in. For tight tasks it can be super helpful -- like for me, an AI/Data Science guy, setting up a basic reverse proxy. But I do so with a ton of scrutiny -- pushing it, searching on Kagi or docs to at least confirm the code, etc. This is helpful because I don't have a mental map about reverse proxy -- so it can help fill in gaps but only with a lot of reticence. That type of use really doesn't justify the billion dollar valuations of any companies, IMO.
- ethanwillis 1y ago
  What do you mean by you don't have a mental map about a reverse proxy?
  - tomrod 1y ago
    That I don't have a deep understanding about nginx and how it's many options fit together across an OS with confidence that I made it secure, accurate, and/or fast. Give me python, Matlab, rust, I can put something together, but something like nginx and I've simply never dived deep enough for a solid understanding before using an LLM to understand more.
caro_kann 1y ago
Even scaffolding a new project is not easy work, especially a new stack or new versions of existing tools. For example, I have never been able to create a Vue 3 project with Vite and Tailwind setup correctly. I tried top SOTA models. Maybe my prompting skills are not good, but everytime it fails to set up a project correctly. Everytime it gives me some old configurations that's not relevant anymore.
- Rapzid 1y ago
  Why though? Vite supplies a project scaffolder and lists a one liner under getting started; ie pnpm create vite Tailwind is similarly a one liner to initialize(might be a vite create option now). Edit: My bad, you are talking about the LLMs! I'm always surprised how still for past years, even though we have great projects scalfolding across the node verse, people are still complaining about how hard setting up projects is..
- lbreakjai 1y ago
  LLMs are probably the worst tool for the job. Code generators have been a thing forever. Why use a LLM when you can do "npm create vite@latest my-vue-app -- --template vue" ?
  - SchemaLoad 1y ago
    It's always more tedious than that. You have to pick all these libraries, install and set them up, build login pages, etc. Stuff that is all simple work but takes ages. I've never used an LLM for it but it seems like the kind of work that should be easy enough to automate and would save a week of setting everything up if it worked.
rkozik1989 1y ago
The reason LLMs suck in plenty of brownfield projects is because those codebases likely either implemented frameworks in a proprietary way, maybe did not rely on any public framework at all, or were in general done in an esoteric way and therefor few (if any) similar codebases exist within the LLMs training data. Which is problematic because LLMs aren't capable of reasoning or learning they're literally just predicting the next most likely token in a chain similarly to how autocomplete works. Without you supplying additional context and explicitly defining guardrails for preforming common tasks the LLM has no frame of reference for working with your codebase.
empath75 1y ago
I actually completely disagree with this, and IMO it works best with projects that are templated with AI development in mind. Lots of documentation and comments, working tests, etc. You want as much context as possible _right in the code_.

herpdyderp 1y ago

I've had great success with GPT5 in existing projects because its agent mode is very good (the best I've seen so far) at analyzing the existing codebase and then writing code that feels like it fits in already (without prompt engineering on my part). I still agree that AI is particularly good on fresh projects though.

SchemaLoad 1y ago
Could be that there is a huge difference in the products. Last few companies have given me Github Copilot which I find entirely useless, I found the automatic suggestions more distracting than useful, and the fix and explain functions never work. But maybe if you burn $1000/day on Claude Code it works a lot better. And then companies see the results from that and wonder why they aren't getting it spending a couple of dollars on Copilot.
- herpdyderp 1y ago
  I use GitHub Copilot from work in agent mode with GPT5 and it’s great! I don’t use the suggestions, fix, or explain features, I agree they’re almost always not helpful.
  - SchemaLoad 1y ago
    Is that a separate product you have to pay for? I've never seen an agent feature in the VS code plugin.
    - adithyassekhar 1y ago
      It's there next to the model selector in chat.

tomrod 1y ago

Exactly. It quickly builds a lot of technical debt that must be paid down, especially for people writing code in areas they aren't deep in. For tight tasks it can be super helpful -- like for me, an AI/Data Science guy, setting up a basic reverse proxy. But I do so with a ton of scrutiny -- pushing it, searching on Kagi or docs to at least confirm the code, etc. This is helpful because I don't have a mental map about reverse proxy -- so it can help fill in gaps but only with a lot of reticence. That type of use really doesn't justify the billion dollar valuations of any companies, IMO.

ethanwillis 1y ago
What do you mean by you don't have a mental map about a reverse proxy?
- tomrod 1y ago
  That I don't have a deep understanding about nginx and how it's many options fit together across an OS with confidence that I made it secure, accurate, and/or fast. Give me python, Matlab, rust, I can put something together, but something like nginx and I've simply never dived deep enough for a solid understanding before using an LLM to understand more.

caro_kann 1y ago

Even scaffolding a new project is not easy work, especially a new stack or new versions of existing tools. For example, I have never been able to create a Vue 3 project with Vite and Tailwind setup correctly. I tried top SOTA models. Maybe my prompting skills are not good, but everytime it fails to set up a project correctly. Everytime it gives me some old configurations that's not relevant anymore.

Rapzid 1y ago
Why though? Vite supplies a project scaffolder and lists a one liner under getting started; ie pnpm create vite Tailwind is similarly a one liner to initialize(might be a vite create option now). Edit: My bad, you are talking about the LLMs! I'm always surprised how still for past years, even though we have great projects scalfolding across the node verse, people are still complaining about how hard setting up projects is..
lbreakjai 1y ago
LLMs are probably the worst tool for the job. Code generators have been a thing forever. Why use a LLM when you can do "npm create vite@latest my-vue-app -- --template vue" ?
- SchemaLoad 1y ago
  It's always more tedious than that. You have to pick all these libraries, install and set them up, build login pages, etc. Stuff that is all simple work but takes ages. I've never used an LLM for it but it seems like the kind of work that should be easy enough to automate and would save a week of setting everything up if it worked.

rkozik1989 1y ago

The reason LLMs suck in plenty of brownfield projects is because those codebases likely either implemented frameworks in a proprietary way, maybe did not rely on any public framework at all, or were in general done in an esoteric way and therefor few (if any) similar codebases exist within the LLMs training data. Which is problematic because LLMs aren't capable of reasoning or learning they're literally just predicting the next most likely token in a chain similarly to how autocomplete works. Without you supplying additional context and explicitly defining guardrails for preforming common tasks the LLM has no frame of reference for working with your codebase.

empath75 1y ago

I actually completely disagree with this, and IMO it works best with projects that are templated with AI development in mind. Lots of documentation and comments, working tests, etc. You want as much context as possible _right in the code_.