11 ms·

> it's not clear if the value actually exists here. Having spent a couple of weeks on Claude Code recently, I arrived to the conclusion that the net value for

> it's not clear if the value actually exists here. Having spent a couple of weeks on Claude Code recently, I arrived to the conclusion that the net value for me from agentic AI is actually negative. I will give it another run in 6-8 months though.

wahnfrieden 1y ago
Did you try with using Opus exclusively?
- freedomben 1y ago
  Do you know if there's a way to force Claude code to do that exclusively? I've found a few env vars online but they don't seem to actually work
  - wahnfrieden 1y ago
    Peter Steinberger has been documenting his workflows and he relies exclusively on Opus at least until recently. (He also pays for a few Max 20x subscriptions at once to avoid rate limits.)
  - atonse 1y ago
    You can type /config and then go to the setting to pick a model.
  - gdudeman 1y ago
    Yes: type /model and then pick Opus 4.1.
  - artursapek 1y ago
    You can "force" it by just paying them $200 (which is nothing compared to the value)
    - parineum 1y ago
      Value is irrelevant. What's the return on investment you get from spending $200? Collecting value doesn't really get you anywhere if nobody is compensating you for it. Unless someone is going to either pay for it for you or give you $200/mo post-tax dollars, it's costing you money.
      - wahnfrieden 1y ago
        The return for me is faster output of features, fixes, and polish for my products which increases revenue above the cost of the tool. Did you need to ask this?
        parineum 1y ago
        Yes, I did. Not everybody has their own product that might benefit from a $200 subscription. Most of us work for someone else and, unless that person is paying for the subscription, the _value_ it adds is irrelevant unless it results in better compensation. Furthermore, the advice was given to upgrade to a $200 subscription from the $20 subscription. The difference in value that might translate into income between the $20 option and the $200 option is very unclear.
        wahnfrieden 1y ago
        If you are employed you should petition your employer for tools you want. Maybe you can use it to take the day off earlier or spend more time socializing. Or to get a promotion or performance bonus. Hopefully not just to meet rising productivity expectations without being handed the tools needed to achieve that. Having full-time access to these tools can also improve your own skills in using them, to profit from in a later career move or from contributing toward your own ends.
        parineum 1y ago
        I'm not disputing that. I'm just pushing back against the casual suggestion (not by you) to just go spend $200. No doubt that you should ask you employer for the tools you want/need to do your job but plenty of us are using this kind of thing casually and the response to "Any way I can force it to use [Opus] exclusively?" is "Spend $200, it's worth it." isn't really helpful, especially in the context where the poster was clearly looking to try it out to see if it was worth it.
mark_l_watson 1y ago
I am sort of with you. I am down to asking Gemini Pro a couple of questions a day, use ChatGPT just a few times a week, and about once a week use gemini-cli (either a short free session, or a longer session where I provide my API key.) That said I spend (waste?) an absurdly large amount of time each week experimenting with local models (sometimes practical applications, sometimes ‘research’).
mikepurvis 1y ago
For a bit more nuance, I think I would my overall net is about break even. But I don't take that as "it's not worth it at all, abandon ship" but rather that I need to hone my instinct of what is and is not a good task for AI involvement, and what that involvement should look like. Throwing together a GHA workflow? Sure, make a ticket, assign it to copilot, check in later to give a little feedback and we're golden. Half a day of labour turned into fifteen minutes. But there are a lot of tasks that are far too nuanced where trying to take that approach just results in frustration and wasted time. There it's better to rely on editor completion or maybe the chat interface, like "hey I want to do X and Y, what approach makes sense for this?" and treat it like a rubber duck session with a junior colleague.
cambaceres 1y ago
For me it’s meant a huge increase in productivity, at least 3X. Since so many claim the opposite, I’m curious to what you do more specifically? I guess different roles/technologies benefit more from agents than others. I build full stack web applications in node/.net/react, more importantly (I think) is that I work on a small startup and manage 3 applications myself.
- datadrivenangel 1y ago
  How do you structure your applications for maintainability?
- dingnuts 1y ago
  You have small applications following extremely common patterns and using common libraries. Models are good at regurgitating patterns they've seen many times, with fuzzy find/replace translations applied. Try to build something like Kubernetes from the ground up and let us know how it goes. Or try writing a custom firmware for a device you just designed. Something like that.
- elevatortrim 1y ago
  I think there are two broad cases where ai coding is beneficial: 1. You are a good coder but working on a new (to you) or building a new project, or working with a technology you are not familiar with. This is where AI is hugely beneficial. It does not only accelerate you, it lets you do things you could not otherwise. 2. You have spent a lot of time on engineering your context and learning what AI is good at, and using it very strategically where you know it will save time and not bother otherwise. If you are a really good coder, really familiar with the project, and mostly changing its bits and pieces rather than building new functionality, AI won’t accelerate you much. Especially if you did not invest the time to make it work well.
- nicce 1y ago
  > I build full stack web applications in node/.net/react, more importantly (I think) is that I work on a small startup and manage 3 applications myself. I think this is your answer. For example, React and JavaScript are extremely popular and aged. Are you using TypeScript and want to get most of the types or are you accepting everything that LLM gives as JavaScript? How interested you are about the code whether it is using "soon to be deprecated" functions or the most optimized loop/implementation? How about the project structure? In other cases, the more precision you need, less effective LLM is.
revskill 1y ago
Truth. To some extend, the agent doesn't know what it's doing at all, it lacks real brain, maybe we should just treat them as the hard worker.
flowerthoughts 1y ago
What type of work do you do? And how do you measure value? Last week I was using Claude Code for web development. This week, I used it to write ESP32 firmware and a Linux kernel driver. Sure, it made mistakes, but the net was still very positive in terms of efficiency.
- verall 1y ago
  > This week, I used it to write ESP32 firmware and a Linux kernel driver. I'm not meaning to be negative at all, but was this for a toy/hobby or for a commercial project? I find that LLMs do very well on small greenfield toy/hobby projects but basically fall over when brought into commercial projects that often have bespoke requirements and standards (i.e. has to cross compile on qcc, comply with autosar, in-house build system, tons of legacy code laying around maybe maybe not used). So no shade - I'm just really curious what kind of project you were able get such good results writing ESP32 FW and kernel drivers for :)
  - lukebechtel 1y ago
    Maintaining project documentation is: (1) Easier with AI (2) Critical for letting AI work effectively in your codebase. Try creating well structured rules for working in your codebase, put in .cursorrules or Claude equivalent... let AI help you... see if that helps.
    - theshrike79 1y ago
      The magic to using agentic LLMs efficiently is... proper project management. You need to have good documentation, split into logical bits. Tasks need to be clearly defined and not have extensive dependencies. And you need to have a simple feedback loop where you can easily run the program and confirm the output matches what you want.
    - troupo 1y ago
      And the chance of that working depends on the weather, the phase of the moon and the arrangement of bird bones in a druidic augury. It's a non-deterministic system producing statistically relevant results with no failure modes. I had Cursor one-shot issues in internal libraries with zero rules. And then suggest I use StringBuilder (Java) in a 100% Elixir project with carefully curated cursor rules as suggested by the latest shamanic ritual trends.
greenie_beans 1y ago
same. agents are good with easy stuff and debugging but extremely bad with complexity. has no clue about chesterson's fence, and it's hard to parse the results especially when it creates massive diffs. creates a ton of abandoned/cargo code. lots of misdirection with OOP. chatting witch claude and copy/pasting code between my IDE and claude is still the most effective for more complex stuff, at least for me.
jmartrican 1y ago
Maybe that is a skills issue.
- rootusrootus 1y ago
  If you are suggesting that LLMs are proving quite good at taking over the low skilled work that probably 90% of devs spend the majority of their time doing, I totally agree. It is the simplest explanation for why many people think they are magic, while some people find very little value. On the occasion that I find myself having to write web code for whatever reason, I'm very happy to have Claude. I don't enjoy coding for the web, like at all.
  - logicprog 1y ago
    I think that's definitely true — these tools are only really taking care of the relatively low skill stuff; synthesizing algorithms and architectures and approaches that have been seen before, automating building out for scaffolding things, or interpolating skeletons, and running relatively typical bash commands for you after making code changes, or implementing fairly specific specifications of how to approach novel architectures algorithms or code logic, automating exploring code bases and building understanding of what things do and where they are and how they relate and the control flow (which would otherwise take hours of laboriously grepping around and reading code), all in small bite sized pieces with a human in the loop. They're even able to make complete and fully working code for things that are a small variation or synthesization of things they've seen a lot before in technologies they're familiar with. But I think that that can still be a pretty good boost — I'd say maybe 20 to 30%, plus MUCH less headache, when used right — even for people that are doing really interesting and novel things, because even if your work has a lot of novelty and domain knowledge to it, there's always mundane horseshit that eats up way too much of your time and brain cycles. So you can use these agents to take care of all the peripheral stuff for you and just focus on what's interesting to you. Imagine you want to write some really novel unique complex algorithm or something but you do want it to have a GUI debugging interface. You can just use Imgui or TKinter if you can make Python bindings or something and then offload that whole thing onto the LLM instead of having to have that extra cognitive load and have to page just to warp the meat of what you're working on out whenever you need to make a modification to your GUI that's more than trivial. I also think this opens up the possibility for a lot more people to write ad hoc personal programs for various things they need, which is even more powerful when combined with something like Python that has a ton of pre-made libraries that do all the difficult stuff for you, or something like emacs that's highly malleable and rewards being able to write programs with it by making them able to very powerfully integrate with your workflow and environment. Even for people who already know how to program and like programming even, there's still an opportunity cost and an amount of time and effort and cognitive load investment in making programs. So by significantly lowering that you open up the opportunities even for us and for people who don't know how to program at all, their productivity basically goes from zero to one, an improvement of 100% (or infinity lol)
ericmcer 1y ago
Agreed, daily Cursor user. Just got out of a 15m huddle with someone trying to understand what they were doing in a PR before they admitted Claude generated everything and it worked but they weren't sure why... Ended up ripping about 200 LoC out because what Claude "fixed" wasn't even broken. So never let it generate code, but the autocomplete is absolutely killer. If you understand how to code in 2+ languages you can make assumptions about how to do things in many others and let the AI autofill the syntax in. I have been able to swap to languages I have almost no experience in and work fairly well because memorizing syntax is irrelevant.
- daymanstep 1y ago
  > I have been able to swap to languages I have almost no experience in and work fairly well because memorizing syntax is irrelevant. I do wonder whether your code does what you think it does. Similar-sounding keywords in different languages can have completely different meanings. E.g. the volatile keyword in Java vs C++. You don't know what you don't know, right? How do you know that the AI generated code does what you think it does?
  - jacobr1 1y ago
    Beyond code-gen I think some techniques are very underutilized. One can generate tests, generate docs, explain things line by line. Explicitly explaining alternative approaches and tradeoffs is helpful too. While, as with everything in this space, there are imperfection, I find a ton of value in looking beyond the code into thinking through the use cases, alternative approaches and different ways to structure the same thing.
    - pornel 1y ago
      I've wasted time debugging phantom issues due to LLM-generated tests that were misusing an API. Brainstorming/explanations can be helpful, but also watch out for Gell-Mann amnesia. It's annoying that LLMs always sound smart whether they are saying something smart or not.
meowtimemania 1y ago
For me it depends on the task. For some tasks (maybe things that don't have good existing examples in my codebase?) I'll spend 3x the time repeatedly asking claude to do something for me
9cb14c1ec0 1y ago
The more I use Claude Code, the more aware I become of its limitations. On the whole, it's a useful tool, but the bigger the codebase the less useful. I've noticed a big difference on its performance on projects with 20k lines of code versus 100k. (Yes, I know. A 100k line project is still very small in the big picture)
- Aeolun 1y ago
  I think one of thr big issues with CC is that it’ll read the first occurence of something, and then think it’s found it. Never mind that there are 17 instances spread throughout the codebase. I have to be really vigilant and tell it to search the codebase for any duplication, then resolve it, if I want it to keep going good at what it does.
sorhaindop 1y ago
This exact phrase has been said by 3 different users... weird.
- sorhaindop 1y ago
  "Having spent a couple of weeks on Claude Code recently, I arrived to the conclusion that the net value for me from agentic AI is actually negative" - smells like BS to me.

wahnfrieden 1y ago

Did you try with using Opus exclusively?

freedomben 1y ago
Do you know if there's a way to force Claude code to do that exclusively? I've found a few env vars online but they don't seem to actually work
- wahnfrieden 1y ago
  Peter Steinberger has been documenting his workflows and he relies exclusively on Opus at least until recently. (He also pays for a few Max 20x subscriptions at once to avoid rate limits.)
- atonse 1y ago
  You can type /config and then go to the setting to pick a model.
- gdudeman 1y ago
  Yes: type /model and then pick Opus 4.1.
- artursapek 1y ago
  You can "force" it by just paying them $200 (which is nothing compared to the value)
  - parineum 1y ago
    Value is irrelevant. What's the return on investment you get from spending $200? Collecting value doesn't really get you anywhere if nobody is compensating you for it. Unless someone is going to either pay for it for you or give you $200/mo post-tax dollars, it's costing you money.
    - wahnfrieden 1y ago
      The return for me is faster output of features, fixes, and polish for my products which increases revenue above the cost of the tool. Did you need to ask this?
      - parineum 1y ago
        Yes, I did. Not everybody has their own product that might benefit from a $200 subscription. Most of us work for someone else and, unless that person is paying for the subscription, the _value_ it adds is irrelevant unless it results in better compensation. Furthermore, the advice was given to upgrade to a $200 subscription from the $20 subscription. The difference in value that might translate into income between the $20 option and the $200 option is very unclear.
        wahnfrieden 1y ago
        If you are employed you should petition your employer for tools you want. Maybe you can use it to take the day off earlier or spend more time socializing. Or to get a promotion or performance bonus. Hopefully not just to meet rising productivity expectations without being handed the tools needed to achieve that. Having full-time access to these tools can also improve your own skills in using them, to profit from in a later career move or from contributing toward your own ends.
        parineum 1y ago
        I'm not disputing that. I'm just pushing back against the casual suggestion (not by you) to just go spend $200. No doubt that you should ask you employer for the tools you want/need to do your job but plenty of us are using this kind of thing casually and the response to "Any way I can force it to use [Opus] exclusively?" is "Spend $200, it's worth it." isn't really helpful, especially in the context where the poster was clearly looking to try it out to see if it was worth it.

mark_l_watson 1y ago

I am sort of with you. I am down to asking Gemini Pro a couple of questions a day, use ChatGPT just a few times a week, and about once a week use gemini-cli (either a short free session, or a longer session where I provide my API key.) That said I spend (waste?) an absurdly large amount of time each week experimenting with local models (sometimes practical applications, sometimes ‘research’).

mikepurvis 1y ago

For a bit more nuance, I think I would my overall net is about break even. But I don't take that as "it's not worth it at all, abandon ship" but rather that I need to hone my instinct of what is and is not a good task for AI involvement, and what that involvement should look like. Throwing together a GHA workflow? Sure, make a ticket, assign it to copilot, check in later to give a little feedback and we're golden. Half a day of labour turned into fifteen minutes. But there are a lot of tasks that are far too nuanced where trying to take that approach just results in frustration and wasted time. There it's better to rely on editor completion or maybe the chat interface, like "hey I want to do X and Y, what approach makes sense for this?" and treat it like a rubber duck session with a junior colleague.

cambaceres 1y ago

For me it’s meant a huge increase in productivity, at least 3X. Since so many claim the opposite, I’m curious to what you do more specifically? I guess different roles/technologies benefit more from agents than others. I build full stack web applications in node/.net/react, more importantly (I think) is that I work on a small startup and manage 3 applications myself.

datadrivenangel 1y ago
How do you structure your applications for maintainability?
dingnuts 1y ago
You have small applications following extremely common patterns and using common libraries. Models are good at regurgitating patterns they've seen many times, with fuzzy find/replace translations applied. Try to build something like Kubernetes from the ground up and let us know how it goes. Or try writing a custom firmware for a device you just designed. Something like that.
elevatortrim 1y ago
I think there are two broad cases where ai coding is beneficial: 1. You are a good coder but working on a new (to you) or building a new project, or working with a technology you are not familiar with. This is where AI is hugely beneficial. It does not only accelerate you, it lets you do things you could not otherwise. 2. You have spent a lot of time on engineering your context and learning what AI is good at, and using it very strategically where you know it will save time and not bother otherwise. If you are a really good coder, really familiar with the project, and mostly changing its bits and pieces rather than building new functionality, AI won’t accelerate you much. Especially if you did not invest the time to make it work well.
nicce 1y ago
> I build full stack web applications in node/.net/react, more importantly (I think) is that I work on a small startup and manage 3 applications myself. I think this is your answer. For example, React and JavaScript are extremely popular and aged. Are you using TypeScript and want to get most of the types or are you accepting everything that LLM gives as JavaScript? How interested you are about the code whether it is using "soon to be deprecated" functions or the most optimized loop/implementation? How about the project structure? In other cases, the more precision you need, less effective LLM is.

revskill 1y ago

Truth. To some extend, the agent doesn't know what it's doing at all, it lacks real brain, maybe we should just treat them as the hard worker.

flowerthoughts 1y ago

What type of work do you do? And how do you measure value? Last week I was using Claude Code for web development. This week, I used it to write ESP32 firmware and a Linux kernel driver. Sure, it made mistakes, but the net was still very positive in terms of efficiency.

verall 1y ago
> This week, I used it to write ESP32 firmware and a Linux kernel driver. I'm not meaning to be negative at all, but was this for a toy/hobby or for a commercial project? I find that LLMs do very well on small greenfield toy/hobby projects but basically fall over when brought into commercial projects that often have bespoke requirements and standards (i.e. has to cross compile on qcc, comply with autosar, in-house build system, tons of legacy code laying around maybe maybe not used). So no shade - I'm just really curious what kind of project you were able get such good results writing ESP32 FW and kernel drivers for :)
- lukebechtel 1y ago
  Maintaining project documentation is: (1) Easier with AI (2) Critical for letting AI work effectively in your codebase. Try creating well structured rules for working in your codebase, put in .cursorrules or Claude equivalent... let AI help you... see if that helps.
  - theshrike79 1y ago
    The magic to using agentic LLMs efficiently is... proper project management. You need to have good documentation, split into logical bits. Tasks need to be clearly defined and not have extensive dependencies. And you need to have a simple feedback loop where you can easily run the program and confirm the output matches what you want.
  - troupo 1y ago
    And the chance of that working depends on the weather, the phase of the moon and the arrangement of bird bones in a druidic augury. It's a non-deterministic system producing statistically relevant results with no failure modes. I had Cursor one-shot issues in internal libraries with zero rules. And then suggest I use StringBuilder (Java) in a 100% Elixir project with carefully curated cursor rules as suggested by the latest shamanic ritual trends.

greenie_beans 1y ago

same. agents are good with easy stuff and debugging but extremely bad with complexity. has no clue about chesterson's fence, and it's hard to parse the results especially when it creates massive diffs. creates a ton of abandoned/cargo code. lots of misdirection with OOP. chatting witch claude and copy/pasting code between my IDE and claude is still the most effective for more complex stuff, at least for me.

jmartrican 1y ago

Maybe that is a skills issue.

rootusrootus 1y ago
If you are suggesting that LLMs are proving quite good at taking over the low skilled work that probably 90% of devs spend the majority of their time doing, I totally agree. It is the simplest explanation for why many people think they are magic, while some people find very little value. On the occasion that I find myself having to write web code for whatever reason, I'm very happy to have Claude. I don't enjoy coding for the web, like at all.
- logicprog 1y ago
  I think that's definitely true — these tools are only really taking care of the relatively low skill stuff; synthesizing algorithms and architectures and approaches that have been seen before, automating building out for scaffolding things, or interpolating skeletons, and running relatively typical bash commands for you after making code changes, or implementing fairly specific specifications of how to approach novel architectures algorithms or code logic, automating exploring code bases and building understanding of what things do and where they are and how they relate and the control flow (which would otherwise take hours of laboriously grepping around and reading code), all in small bite sized pieces with a human in the loop. They're even able to make complete and fully working code for things that are a small variation or synthesization of things they've seen a lot before in technologies they're familiar with. But I think that that can still be a pretty good boost — I'd say maybe 20 to 30%, plus MUCH less headache, when used right — even for people that are doing really interesting and novel things, because even if your work has a lot of novelty and domain knowledge to it, there's always mundane horseshit that eats up way too much of your time and brain cycles. So you can use these agents to take care of all the peripheral stuff for you and just focus on what's interesting to you. Imagine you want to write some really novel unique complex algorithm or something but you do want it to have a GUI debugging interface. You can just use Imgui or TKinter if you can make Python bindings or something and then offload that whole thing onto the LLM instead of having to have that extra cognitive load and have to page just to warp the meat of what you're working on out whenever you need to make a modification to your GUI that's more than trivial. I also think this opens up the possibility for a lot more people to write ad hoc personal programs for various things they need, which is even more powerful when combined with something like Python that has a ton of pre-made libraries that do all the difficult stuff for you, or something like emacs that's highly malleable and rewards being able to write programs with it by making them able to very powerfully integrate with your workflow and environment. Even for people who already know how to program and like programming even, there's still an opportunity cost and an amount of time and effort and cognitive load investment in making programs. So by significantly lowering that you open up the opportunities even for us and for people who don't know how to program at all, their productivity basically goes from zero to one, an improvement of 100% (or infinity lol)

ericmcer 1y ago

Agreed, daily Cursor user. Just got out of a 15m huddle with someone trying to understand what they were doing in a PR before they admitted Claude generated everything and it worked but they weren't sure why... Ended up ripping about 200 LoC out because what Claude "fixed" wasn't even broken. So never let it generate code, but the autocomplete is absolutely killer. If you understand how to code in 2+ languages you can make assumptions about how to do things in many others and let the AI autofill the syntax in. I have been able to swap to languages I have almost no experience in and work fairly well because memorizing syntax is irrelevant.

daymanstep 1y ago
> I have been able to swap to languages I have almost no experience in and work fairly well because memorizing syntax is irrelevant. I do wonder whether your code does what you think it does. Similar-sounding keywords in different languages can have completely different meanings. E.g. the volatile keyword in Java vs C++. You don't know what you don't know, right? How do you know that the AI generated code does what you think it does?
- jacobr1 1y ago
  Beyond code-gen I think some techniques are very underutilized. One can generate tests, generate docs, explain things line by line. Explicitly explaining alternative approaches and tradeoffs is helpful too. While, as with everything in this space, there are imperfection, I find a ton of value in looking beyond the code into thinking through the use cases, alternative approaches and different ways to structure the same thing.
  - pornel 1y ago
    I've wasted time debugging phantom issues due to LLM-generated tests that were misusing an API. Brainstorming/explanations can be helpful, but also watch out for Gell-Mann amnesia. It's annoying that LLMs always sound smart whether they are saying something smart or not.

meowtimemania 1y ago

For me it depends on the task. For some tasks (maybe things that don't have good existing examples in my codebase?) I'll spend 3x the time repeatedly asking claude to do something for me

9cb14c1ec0 1y ago

The more I use Claude Code, the more aware I become of its limitations. On the whole, it's a useful tool, but the bigger the codebase the less useful. I've noticed a big difference on its performance on projects with 20k lines of code versus 100k. (Yes, I know. A 100k line project is still very small in the big picture)

Aeolun 1y ago
I think one of thr big issues with CC is that it’ll read the first occurence of something, and then think it’s found it. Never mind that there are 17 instances spread throughout the codebase. I have to be really vigilant and tell it to search the codebase for any duplication, then resolve it, if I want it to keep going good at what it does.

sorhaindop 1y ago

This exact phrase has been said by 3 different users... weird.

sorhaindop 1y ago
"Having spent a couple of weeks on Claude Code recently, I arrived to the conclusion that the net value for me from agentic AI is actually negative" - smells like BS to me.