5 ms·

All I hear is anecdotal statements from people claiming LLMs have made them some percent more productive. Yet few actually say how or demonstrate it. For the l

by geuis 2y ago

All I hear is anecdotal statements from people claiming LLMs have made them some percent more productive. Yet few actually say how or demonstrate it. For the last year, I've tried all sorts of models both as hosted services and running locally with llama.cpp or ollama. I've used both the continue.dev vscode extension and cursor more recently. The results have been frustrating at best. The user interface of the tools is just awful. The output of any models from Deepseek to quen to Claude to whatever other model is mediocre to useless. I literally highlight some code that includes comments about what I need and I even include long explicit descriptions etc in the prompts and it's just unrelated garbage out every time. The most useful thing has just been ChatGPT when there's something I need to learn about. Rubber ducking basically. It's alright at very simple coding questions or asking about obscure database questions I might have, but beyond that it's useless. Gotta keep the context window short, or it starts going off the rails every single time.

infecto 2y ago
Just in the past couple months there have been a number of "I am a senior/principal engineer and this is how I use LLMs". I would agree that the tools are not optimal yet but every iteration has improved for me. Maybe whatever language you are coding it or whatever project you are working on is not a good fit? It is an equally perplexing situation for myself when I hear anecdotes like yours which don't align with my experience. The fact that you say everything is garbage calls into question either how you are using the tool or something else. I can reliably use cursor's composer to reference a couple files, give a bullet list of what we are trying to do and point it to one of the better models and the output is junior engineer level or better output. When I say junior, I mean a junior who has experience with the codebase.
- andy24 2y ago
  What kinds of projects are you working on? My experience is jot very good with these tools (expanded in a sibling comment).
  - infecto 2y ago
    Generally a lot of web-dev which is where I would assume LLMs shine the best. I noted elsewhere but I think it depends a lot on the age of the product and the level of velocity. For early life products where the speed of your velocity matters, I think you can get the most benefit. The more mature the product and the slower the team implements features, the benefits are still measurable but not as high.
    - andy24 2y ago
      Ah yeah, I can totally see how it can be useful for churning put tons of code. Even without copy-paste, just generating a ton of references and rewriting/improving them. Anecdotally, I’ve tried asking deepseek to review a few files of my code — it wasn’t bad at all, though not without false positives.
andy24 2y ago
I have a similar experience. Tried to use it for real work and got frustrated by the chat’s inability to say “I don’t know”. It’s okay for code snippets demonstrating how something can be used (stack overflow essentially), also code reviews can be helpful if doing something for the first time. But they fail to answer questions I’m interested in like “what’s the purpose of X”.
ta1243 2y ago
> All I hear is anecdotal statements from people claiming LLMs have made them some percent more productive. Yet few actually say how or demonstrate it. It's very difficult to measure productivity of most people, certainly most people in office jobs, so while you can have a gut feeling that you're doing better, it's no more measurable than pre-AI individual productivity measurement was
- skydhash 2y ago
  It’s not really about objective measurements, but practical applications. Like try this in the following manner and compare it to your previous workflow. Sensible advices like the ones found in The Pragmatic Programmer.
  - ta1243 2y ago
    Sure, so it's always going to be annecdotal. That doesn't mean the benefits don't exist, just means they can't be objectively measured. Just like we can't objectively measure the output of a single knowlege worker, especially output on a single day
alexvitkov 2y ago
If LLM chatbots are making you vastly more productive in a field, you are in the bottom 20% of that field. They're still useful tools for exploring new disciplines, but if you're say a programmer and you think ChatGPT or DeepSeek is good at programming, that's a good sign you need to start improving.
- agentultra 2y ago
  This. I shudder to think of the hubris of a programmer who doesn’t understand pointers prompting an AI model to generate low-level system code for them. Sure it might generate a program that appears to work. But is that human reading the code qualified to review it and prevent the model from generating subtle, non-obvious errors?
- infecto 2y ago
  If you have to tell others that then perhaps some introspection for yourself might be helpful. Comes across more as denial than constructive commentary. I do believe the benefit decreases the more senior or familiar the work is but there is still a noticeable benefit and I think it largely depends on the velocity and life cycle of the product. I think you get less benefit the slower the velocity or the more mature of a product. To deny it like in your post is simply being an intellectual minimalist.
  - nyeah 2y ago
    I see a gap between "vastly more productive" and "noticeable benefit".
    - red-iron-pine 2y ago
      good shoes help me walk a bit faster, and for longer. they don't let me walk at the pace of a SUV. AI is like the good shoes. they help, and make many tasks a bit easier. but they can't make me into an SUV. and if they can, then no programmers will have jobs. which is the end-state of this whole LLM thing as far as I can tell.
deleted 2y ago
[deleted]
strangescript 2y ago
I think its about scope and expectations. I have had some form of AI code completer in my neovim config for 3 years. It works flawlessly and saves me tons of keystrokes. Sure sometimes it suggests the incorrect completion but I just ignore it and keep coding as if it didn't exist. I am talking about line by line, not entire code blocks, but even that it does well at times. From what I have seen the people that have the most success have AI building something from scratch using well known tooling (read: old tooling). The problem is that doesn't immediately help most people. We are all stuck in crap jobs with massive, crusty code bases. Its hard for AI because its hard for everyone.
llm_trw 2y ago
I fixed the hinge in my oven by giving perplexity.com the make and problem. I saved an hour on the phone calling people to organise a visit some time in the next week. Maybe you should stop using the Ai slop tools that don't work?
- sharpshadow 2y ago
  Very likely you would have been as successful without AI to fix your oven hinge yourself, there is tons of content about that online.
  - llm_trw 2y ago
    No. I'd already spend 30 Min looking at how to solve it myself. The search on perplexity was a hail marry before I started calling handymen.
vipshek 2y ago
I would like to propose a moratorium on these sorts of “AI coding is good” or “AI coding sucks” comments without any further context. This comment is like saying, “This diet didn’t work for me” without providing any details about your health circumstances. What’s your weight? Age? Level of activity? In this context: What language are you working in? What frameworks are you using? What’s the nature of your project? How legacy is your codebase? How big is the codebase? If we all outline these factors plus our experiences with these tools, then perhaps we can collectively learn about the circumstances when they work or don’t work. And then maybe we can make them better for the circumstances where they’re currently weak.
- azemetre 2y ago
  I feel like diet as an analogy doesn't work. We know that the only way to lose weight is with a caloric deficit. If you can't do this, it doesn't matter what you eat you won't lose weight. If you're failing to lose weight because of a diet you are eating too much, full stop. Whereas measuring productivity and usefulness is way more opaque. Many simple software systems are highly productive for their companies.
xienze 2y ago
I agree with the other commenter that said if you're "vastly" more productive as a developer due to AI, you probably weren't that good to begin with. Otherwise, please provide concrete examples. Myself, I do find it quite useful in a few respects. First and foremost, as a "better Google/StackOverflow." If something's not working, I can describe my exact scenario and usually get pointed in the right direction. Sometimes the LLM just wastes my time by very confidently telling me some function/library that solves my exact problem exists when in fact it doesn't. Second, IntelliJ's local LLM is sort of a smarter autocomplete. It makes some outright wrong suggestions, but when there's areas where I have to do a lot of repetitive tasks that follow a simple pattern (like for instance, mapping fields from one type of object to another), it does a pretty good job of making correct suggestions. I definitely appreciate it but it's certainly not doing things like writing a significant portion of code in my style.
vel0city 2y ago
I've been using Amazon Q Developer as it was provided and approved by my employer. It has been pretty good with Python codebases, Kubernetes configurations, and (not surprisingly) CDK/Cloudformation templates. I can pretty much just ask it "here's my python script, make everything I need to run it as a lambda, hook that lambda up to x, it should run in a vpc defined in this template over here", and it'll get all that stuff put together and its normally pretty solid code it generates. It seems to pull in a lot of the context of the project I have open. For instance, I can say "it should get those values from the outputs in other-cf-template.yml" and it knows the naming schemes and what not across templates, even if it didn't generate those templates. I might go back and tweak some stuff, add some extra tags and what not, but often its pretty good at doing what I ask. Sometimes its suggestions aren't what I was really wanting to do in my codebase, a handful of times it has made up methods or parameters of even well-known libraries. But usually, its suggestions are better than a basic IntelliSense-style autocomplete at least in my experiences. I haven't used many of the other developer assistant plugins like say GitHub Copilot. I couldn't really say which is better or worse. But I do think using Q Developer has made me faster in many tasks. I wouldn't expect a tool that doesn't have access to the context of my editor and the files I have open to be very useful for actually coding. There's a lot of context to understand in even a basic application. If you're just asking a locally running app in ollama "give me a method to do x", don't be surprised if it doesn't know everything else happening in your app. Maybe it'll give you a halfway decent example of doing something, but devoid of how it actually plugs in to whatever you're making it might be entirely worthless.
F7F7F7 2y ago
Seriously. It’s like half of the people in this thread are living in a completely different world. And this is coming from someone who uses LLMs daily at the subscription, API (vscode and 3 nextjs apps) and local level. I have a custom langchain stack, prompt repo, you name it. And regardless of how little or how much I use what I have, or what soup de jour prompt or process (from Keep it simple to Prompt enhancers) I can’t say it’s made a meaningful difference in my life. Even with all of the customization and planning. It’s a great search engine though.
- Earw0rm 2y ago
  Would it look like such a good search engine if the actual search engines hadn't progressively broken themselves over the last 15 years? I swear half the time when I use it to look up the nuances of system API stuff, it's replaying forum, mailing list or Stackoverflow conversations that Google ought to be able to find but somehow can't.

by geuis 2y ago

infecto 2y ago

Just in the past couple months there have been a number of "I am a senior/principal engineer and this is how I use LLMs". I would agree that the tools are not optimal yet but every iteration has improved for me. Maybe whatever language you are coding it or whatever project you are working on is not a good fit? It is an equally perplexing situation for myself when I hear anecdotes like yours which don't align with my experience. The fact that you say everything is garbage calls into question either how you are using the tool or something else. I can reliably use cursor's composer to reference a couple files, give a bullet list of what we are trying to do and point it to one of the better models and the output is junior engineer level or better output. When I say junior, I mean a junior who has experience with the codebase.

andy24 2y ago
What kinds of projects are you working on? My experience is jot very good with these tools (expanded in a sibling comment).
- infecto 2y ago
  Generally a lot of web-dev which is where I would assume LLMs shine the best. I noted elsewhere but I think it depends a lot on the age of the product and the level of velocity. For early life products where the speed of your velocity matters, I think you can get the most benefit. The more mature the product and the slower the team implements features, the benefits are still measurable but not as high.
  - andy24 2y ago
    Ah yeah, I can totally see how it can be useful for churning put tons of code. Even without copy-paste, just generating a ton of references and rewriting/improving them. Anecdotally, I’ve tried asking deepseek to review a few files of my code — it wasn’t bad at all, though not without false positives.

andy24 2y ago

I have a similar experience. Tried to use it for real work and got frustrated by the chat’s inability to say “I don’t know”. It’s okay for code snippets demonstrating how something can be used (stack overflow essentially), also code reviews can be helpful if doing something for the first time. But they fail to answer questions I’m interested in like “what’s the purpose of X”.

ta1243 2y ago

> All I hear is anecdotal statements from people claiming LLMs have made them some percent more productive. Yet few actually say how or demonstrate it. It's very difficult to measure productivity of most people, certainly most people in office jobs, so while you can have a gut feeling that you're doing better, it's no more measurable than pre-AI individual productivity measurement was

skydhash 2y ago
It’s not really about objective measurements, but practical applications. Like try this in the following manner and compare it to your previous workflow. Sensible advices like the ones found in The Pragmatic Programmer.
- ta1243 2y ago
  Sure, so it's always going to be annecdotal. That doesn't mean the benefits don't exist, just means they can't be objectively measured. Just like we can't objectively measure the output of a single knowlege worker, especially output on a single day

alexvitkov 2y ago

If LLM chatbots are making you vastly more productive in a field, you are in the bottom 20% of that field. They're still useful tools for exploring new disciplines, but if you're say a programmer and you think ChatGPT or DeepSeek is good at programming, that's a good sign you need to start improving.

agentultra 2y ago
This. I shudder to think of the hubris of a programmer who doesn’t understand pointers prompting an AI model to generate low-level system code for them. Sure it might generate a program that appears to work. But is that human reading the code qualified to review it and prevent the model from generating subtle, non-obvious errors?
infecto 2y ago
If you have to tell others that then perhaps some introspection for yourself might be helpful. Comes across more as denial than constructive commentary. I do believe the benefit decreases the more senior or familiar the work is but there is still a noticeable benefit and I think it largely depends on the velocity and life cycle of the product. I think you get less benefit the slower the velocity or the more mature of a product. To deny it like in your post is simply being an intellectual minimalist.
- nyeah 2y ago
  I see a gap between "vastly more productive" and "noticeable benefit".
  - red-iron-pine 2y ago
    good shoes help me walk a bit faster, and for longer. they don't let me walk at the pace of a SUV. AI is like the good shoes. they help, and make many tasks a bit easier. but they can't make me into an SUV. and if they can, then no programmers will have jobs. which is the end-state of this whole LLM thing as far as I can tell.

deleted 2y ago

[deleted]

strangescript 2y ago

I think its about scope and expectations. I have had some form of AI code completer in my neovim config for 3 years. It works flawlessly and saves me tons of keystrokes. Sure sometimes it suggests the incorrect completion but I just ignore it and keep coding as if it didn't exist. I am talking about line by line, not entire code blocks, but even that it does well at times. From what I have seen the people that have the most success have AI building something from scratch using well known tooling (read: old tooling). The problem is that doesn't immediately help most people. We are all stuck in crap jobs with massive, crusty code bases. Its hard for AI because its hard for everyone.

llm_trw 2y ago

I fixed the hinge in my oven by giving perplexity.com the make and problem. I saved an hour on the phone calling people to organise a visit some time in the next week. Maybe you should stop using the Ai slop tools that don't work?

sharpshadow 2y ago
Very likely you would have been as successful without AI to fix your oven hinge yourself, there is tons of content about that online.
- llm_trw 2y ago
  No. I'd already spend 30 Min looking at how to solve it myself. The search on perplexity was a hail marry before I started calling handymen.

vipshek 2y ago

I would like to propose a moratorium on these sorts of “AI coding is good” or “AI coding sucks” comments without any further context. This comment is like saying, “This diet didn’t work for me” without providing any details about your health circumstances. What’s your weight? Age? Level of activity? In this context: What language are you working in? What frameworks are you using? What’s the nature of your project? How legacy is your codebase? How big is the codebase? If we all outline these factors plus our experiences with these tools, then perhaps we can collectively learn about the circumstances when they work or don’t work. And then maybe we can make them better for the circumstances where they’re currently weak.

azemetre 2y ago
I feel like diet as an analogy doesn't work. We know that the only way to lose weight is with a caloric deficit. If you can't do this, it doesn't matter what you eat you won't lose weight. If you're failing to lose weight because of a diet you are eating too much, full stop. Whereas measuring productivity and usefulness is way more opaque. Many simple software systems are highly productive for their companies.

xienze 2y ago

I agree with the other commenter that said if you're "vastly" more productive as a developer due to AI, you probably weren't that good to begin with. Otherwise, please provide concrete examples. Myself, I do find it quite useful in a few respects. First and foremost, as a "better Google/StackOverflow." If something's not working, I can describe my exact scenario and usually get pointed in the right direction. Sometimes the LLM just wastes my time by very confidently telling me some function/library that solves my exact problem exists when in fact it doesn't. Second, IntelliJ's local LLM is sort of a smarter autocomplete. It makes some outright wrong suggestions, but when there's areas where I have to do a lot of repetitive tasks that follow a simple pattern (like for instance, mapping fields from one type of object to another), it does a pretty good job of making correct suggestions. I definitely appreciate it but it's certainly not doing things like writing a significant portion of code in my style.

vel0city 2y ago

I've been using Amazon Q Developer as it was provided and approved by my employer. It has been pretty good with Python codebases, Kubernetes configurations, and (not surprisingly) CDK/Cloudformation templates. I can pretty much just ask it "here's my python script, make everything I need to run it as a lambda, hook that lambda up to x, it should run in a vpc defined in this template over here", and it'll get all that stuff put together and its normally pretty solid code it generates. It seems to pull in a lot of the context of the project I have open. For instance, I can say "it should get those values from the outputs in other-cf-template.yml" and it knows the naming schemes and what not across templates, even if it didn't generate those templates. I might go back and tweak some stuff, add some extra tags and what not, but often its pretty good at doing what I ask. Sometimes its suggestions aren't what I was really wanting to do in my codebase, a handful of times it has made up methods or parameters of even well-known libraries. But usually, its suggestions are better than a basic IntelliSense-style autocomplete at least in my experiences. I haven't used many of the other developer assistant plugins like say GitHub Copilot. I couldn't really say which is better or worse. But I do think using Q Developer has made me faster in many tasks. I wouldn't expect a tool that doesn't have access to the context of my editor and the files I have open to be very useful for actually coding. There's a lot of context to understand in even a basic application. If you're just asking a locally running app in ollama "give me a method to do x", don't be surprised if it doesn't know everything else happening in your app. Maybe it'll give you a halfway decent example of doing something, but devoid of how it actually plugs in to whatever you're making it might be entirely worthless.

F7F7F7 2y ago

Seriously. It’s like half of the people in this thread are living in a completely different world. And this is coming from someone who uses LLMs daily at the subscription, API (vscode and 3 nextjs apps) and local level. I have a custom langchain stack, prompt repo, you name it. And regardless of how little or how much I use what I have, or what soup de jour prompt or process (from Keep it simple to Prompt enhancers) I can’t say it’s made a meaningful difference in my life. Even with all of the customization and planning. It’s a great search engine though.

Earw0rm 2y ago
Would it look like such a good search engine if the actual search engines hadn't progressively broken themselves over the last 15 years? I swear half the time when I use it to look up the nuances of system API stuff, it's replaying forum, mailing list or Stackoverflow conversations that Google ought to be able to find but somehow can't.