4 ms·

Also... if your workflow is "think hard, let LLM write it, read what AI wrote, think hard about what AI wrote, re-type what AI wrote, fix what AI wrote"... Whe

Also... if your workflow is "think hard, let LLM write it, read what AI wrote, think hard about what AI wrote, re-type what AI wrote, fix what AI wrote"... Where in the heck are the efficiency gains? Couldn't you just drop the LLM part of it and save the company a trillion dollars in tokens?

ncallaway 2mo ago
I've found I can get quite a bit of use out of it as a basically ever-present rubber duck. Using a cheap model, I usually spend < $1-$2 per day (on API billing) using it a model like. - here's the problem I'm trying to solve: - here's my initial plan for the design: - is there anything I'm not thinking about or my plan is missing? It'll occasionally pop out a suggestion that I like more than my original plan, or it'll give me some new angles to think through the original plan.
- baliex 2mo ago
  I’m guilty of using this approach too and noticed that an (obvious?) side-effect is that I get fewer wonderful, human, “aha” moments with my colleagues. I mentioned this at a stand-up and only then did others think about it and notice the same of themselves. This, to me, is one of the major downsides of all this: less collaboration, more individualism
  - noduerme 2mo ago
    I think that's true. One caveat: If you're a solo coder like me, you almost never get to talk about implementation details with anyone who (a) understands or (b) gives a flying fuck. I'm more likely to have a meaningful convo about code structure with someone on a laptop sitting next to me at the coffeehouse. So sometimes a stand-in bot to bounce your ideas off of can illuminate corners you're missing, even if it's in some ways no operating just as a chat with yourself.
  - bigbuppo 2mo ago
    > This, to me, is one of the major downsides of all this: less collaboration, more individualism Or put another way, it's replacing collaboration and human connection with your team with a subscription service that isolates you, makes you emotionally and cognitively dependent on the service while making your performance numbers look good. I really don't like the sound of that.
- jgord 2mo ago
  I worry about the finger-on-the-scale mean-reversion aspect of this .. the LLM corralling you ever so gently back towards the median. otoh, LLMs really could become the great illustrated primer for educating kids in science / history / languages etc.
  - Always42 2mo ago
    Isn't median code often a good thing in terms of ability for others to understand it?
    - mcosta 2mo ago
      Well, the objetive right now seems to be other LLMs understand it.
    - b345 2mo ago
      Yes, but also not necessarily good code(according to some). A lot of people don't buy into the "code should be optimised for reading" stuff.
cavoirom 2mo ago
knowledge, by doing that I retrieve the knowledge from LLM, I learn faster, write code faster, do things more careful... The efficiency is the better design, better code quality, higher test coverage, less bugs. Because LLM let me gain knowledge faster than previous tools.
crossroadsguy 2mo ago
Besides, many companies proactively punish doing anything not by LLM.
- deleted 2mo ago
  [deleted]
p1esk 2mo ago
I stopped writing code completely about a year ago, stopped reading code completely about 8 months ago, and I feel like I stopped thinking hard about anything at work about 6 months ago. To clarify - I still produce code for a living - all day, every day. Not sure about efficiency gains but I got promoted 3 weeks ago :)
20k 2mo ago
LLM speedups have always been predicated on not understanding the code they produce as well as if you wrote it yourself. Reviewing even well written code to understand it in depth, has always taken longer than simply writing it yourself - let alone sloppy LLM code that you have to fix People are, in heavy LLM systems, realising that the most valuable commodity is engineers knowing WTF is going on, and that loss of understanding of your codebase is the #1 blocker to actually getting things done. Any senior engineer knows this all too well Its why I suspect we never tend to see any longer term LLM productivity stats being published, and why I extra suspect that LLMs have failed to achieve any kind of penetration into open source code. If you take away the pressure to produce bad code, writing by hand clearly wins massively in the long term. Its classic short term gains for a long term penalty
- default-kramer 2mo ago
  > the most valuable commodity is engineers knowing WTF is going on, and that loss of understanding of your codebase is the #1 blocker to actually getting things done Perfectly said. It was true before LLMs too. As I've been reviewing more and more LLM code, I've noticed that while good developers can use LLMs to produce good code maybe 1x-5x faster (depending on circumstances), bad developers can use LLMs to produce bad code 1000x faster. And if you are collecting metrics that only capture that 1000x number, I bet you feel great about what you're doing.
- locknitpicker 2mo ago
  > LLM speedups have always been predicated on not understanding the code they produce as well as if you wrote it yourself. Reviewing even well written code to understand it in depth, has always taken longer than simply writing it yourself - let alone sloppy LLM code that you have to fix I don't think this personal opinion holds well when contrasted with reality. The truth of the matter is that in moderately large codebases, which is the norm in any production setting, at best you have a T-shape understanding of the system: in specific areas you might have somewhat deep understanding, but for the vast majority of the system you have at best a high level understanding of the software architecture. This is why software engineering frames things like design patterns, principle of least surprise,single responsibility, etc as premium design and code quality traits: they allow developers to effectively extrapolate their high level understanding of the project onto components they know nothing about. In any software development setting,you will find that the most successful and even senior engineer is that which successfully and skillfully manages this uncertainty, and is able to hit the ground running on environments they never touched. I had a colleague who once was a senior software development engineer at a FANG that mastered this art, and called it JIT onboarding. So why are you pretending this is something new or novel? I think this blend of criticism only serves to allow inexperienced developers to stand out by complaining that a very mundane aspect of working on projects as a member of a sizeable team is somehow a novel development caused by LLMs. And don't get me started on the "slop" nonsense. Have you actually browsed through code written by people?
devsda 2mo ago
> Where in the heck are the efficiency gains? It has been the opposite for us. It is now being weaponized by PMs and leads to add more work for developers and not less in terms of documentation and sign-offs. Now stories automatically have additional 10 subtasks for documentation and tracking. These are created automatically through "skills" and the task of detailed documentation and updating those extra tasks is a chore. When complained, they just say "use AI". Every JIRA task these days feels like submitting a petition to the government bureaucracy. PMs are happy because they did something new, leaders are happy for more tracking, developers are expected to be happy with "productivity gains" due to not having to write much code.
locknitpicker 2mo ago
> Where in the heck are the efficiency gains? I think there's some confusion in your post. You're seeing people claiming that the rate their code changes is so fast that they find it hard to keep their mental model up to date and in sync with the actual codebase. They frame this as a LLM issue, but those of us who worked in large teams will understand how it feels to be faced with a dozen changes popping up in critical areas of a code base each time we prepare to submit our own code change. Once you understand this, you'll easily understand where all the efficiency gains are manifesting. Today's lone cowboy developers are experiencing the same type of struggles to keep up with a project that in the recent past affected projects worked on by large teams. I mean, think about it for a second: what do you think is behind this higher rate of change?
akd 2mo ago
The LLM is, in many ways, smarter than you and I: it can perform feats of advanced mathematics that have eluded mathematicians for decades. It's not at all unreasonable to think that an LLM could produce an idea better than you could have, but that reading it and typing it in gives you some better understanding of how it works.
nicman23 2mo ago
in code you do not care enough to read, in experimental code, one off things, tests, reading a codebase with no documentation, re jobs

ncallaway 2mo ago

I've found I can get quite a bit of use out of it as a basically ever-present rubber duck. Using a cheap model, I usually spend < $1-$2 per day (on API billing) using it a model like. - here's the problem I'm trying to solve: - here's my initial plan for the design: - is there anything I'm not thinking about or my plan is missing? It'll occasionally pop out a suggestion that I like more than my original plan, or it'll give me some new angles to think through the original plan.

baliex 2mo ago
I’m guilty of using this approach too and noticed that an (obvious?) side-effect is that I get fewer wonderful, human, “aha” moments with my colleagues. I mentioned this at a stand-up and only then did others think about it and notice the same of themselves. This, to me, is one of the major downsides of all this: less collaboration, more individualism
- noduerme 2mo ago
  I think that's true. One caveat: If you're a solo coder like me, you almost never get to talk about implementation details with anyone who (a) understands or (b) gives a flying fuck. I'm more likely to have a meaningful convo about code structure with someone on a laptop sitting next to me at the coffeehouse. So sometimes a stand-in bot to bounce your ideas off of can illuminate corners you're missing, even if it's in some ways no operating just as a chat with yourself.
- bigbuppo 2mo ago
  > This, to me, is one of the major downsides of all this: less collaboration, more individualism Or put another way, it's replacing collaboration and human connection with your team with a subscription service that isolates you, makes you emotionally and cognitively dependent on the service while making your performance numbers look good. I really don't like the sound of that.
jgord 2mo ago
I worry about the finger-on-the-scale mean-reversion aspect of this .. the LLM corralling you ever so gently back towards the median. otoh, LLMs really could become the great illustrated primer for educating kids in science / history / languages etc.
- Always42 2mo ago
  Isn't median code often a good thing in terms of ability for others to understand it?
  - mcosta 2mo ago
    Well, the objetive right now seems to be other LLMs understand it.
  - b345 2mo ago
    Yes, but also not necessarily good code(according to some). A lot of people don't buy into the "code should be optimised for reading" stuff.

cavoirom 2mo ago

knowledge, by doing that I retrieve the knowledge from LLM, I learn faster, write code faster, do things more careful... The efficiency is the better design, better code quality, higher test coverage, less bugs. Because LLM let me gain knowledge faster than previous tools.

crossroadsguy 2mo ago

Besides, many companies proactively punish doing anything not by LLM.

deleted 2mo ago
[deleted]

p1esk 2mo ago

I stopped writing code completely about a year ago, stopped reading code completely about 8 months ago, and I feel like I stopped thinking hard about anything at work about 6 months ago. To clarify - I still produce code for a living - all day, every day. Not sure about efficiency gains but I got promoted 3 weeks ago :)

20k 2mo ago

LLM speedups have always been predicated on not understanding the code they produce as well as if you wrote it yourself. Reviewing even well written code to understand it in depth, has always taken longer than simply writing it yourself - let alone sloppy LLM code that you have to fix People are, in heavy LLM systems, realising that the most valuable commodity is engineers knowing WTF is going on, and that loss of understanding of your codebase is the #1 blocker to actually getting things done. Any senior engineer knows this all too well Its why I suspect we never tend to see any longer term LLM productivity stats being published, and why I extra suspect that LLMs have failed to achieve any kind of penetration into open source code. If you take away the pressure to produce bad code, writing by hand clearly wins massively in the long term. Its classic short term gains for a long term penalty

default-kramer 2mo ago
> the most valuable commodity is engineers knowing WTF is going on, and that loss of understanding of your codebase is the #1 blocker to actually getting things done Perfectly said. It was true before LLMs too. As I've been reviewing more and more LLM code, I've noticed that while good developers can use LLMs to produce good code maybe 1x-5x faster (depending on circumstances), bad developers can use LLMs to produce bad code 1000x faster. And if you are collecting metrics that only capture that 1000x number, I bet you feel great about what you're doing.
locknitpicker 2mo ago
> LLM speedups have always been predicated on not understanding the code they produce as well as if you wrote it yourself. Reviewing even well written code to understand it in depth, has always taken longer than simply writing it yourself - let alone sloppy LLM code that you have to fix I don't think this personal opinion holds well when contrasted with reality. The truth of the matter is that in moderately large codebases, which is the norm in any production setting, at best you have a T-shape understanding of the system: in specific areas you might have somewhat deep understanding, but for the vast majority of the system you have at best a high level understanding of the software architecture. This is why software engineering frames things like design patterns, principle of least surprise,single responsibility, etc as premium design and code quality traits: they allow developers to effectively extrapolate their high level understanding of the project onto components they know nothing about. In any software development setting,you will find that the most successful and even senior engineer is that which successfully and skillfully manages this uncertainty, and is able to hit the ground running on environments they never touched. I had a colleague who once was a senior software development engineer at a FANG that mastered this art, and called it JIT onboarding. So why are you pretending this is something new or novel? I think this blend of criticism only serves to allow inexperienced developers to stand out by complaining that a very mundane aspect of working on projects as a member of a sizeable team is somehow a novel development caused by LLMs. And don't get me started on the "slop" nonsense. Have you actually browsed through code written by people?

devsda 2mo ago

> Where in the heck are the efficiency gains? It has been the opposite for us. It is now being weaponized by PMs and leads to add more work for developers and not less in terms of documentation and sign-offs. Now stories automatically have additional 10 subtasks for documentation and tracking. These are created automatically through "skills" and the task of detailed documentation and updating those extra tasks is a chore. When complained, they just say "use AI". Every JIRA task these days feels like submitting a petition to the government bureaucracy. PMs are happy because they did something new, leaders are happy for more tracking, developers are expected to be happy with "productivity gains" due to not having to write much code.

locknitpicker 2mo ago

> Where in the heck are the efficiency gains? I think there's some confusion in your post. You're seeing people claiming that the rate their code changes is so fast that they find it hard to keep their mental model up to date and in sync with the actual codebase. They frame this as a LLM issue, but those of us who worked in large teams will understand how it feels to be faced with a dozen changes popping up in critical areas of a code base each time we prepare to submit our own code change. Once you understand this, you'll easily understand where all the efficiency gains are manifesting. Today's lone cowboy developers are experiencing the same type of struggles to keep up with a project that in the recent past affected projects worked on by large teams. I mean, think about it for a second: what do you think is behind this higher rate of change?

akd 2mo ago

The LLM is, in many ways, smarter than you and I: it can perform feats of advanced mathematics that have eluded mathematicians for decades. It's not at all unreasonable to think that an LLM could produce an idea better than you could have, but that reading it and typing it in gives you some better understanding of how it works.

nicman23 2mo ago

in code you do not care enough to read, in experimental code, one off things, tests, reading a codebase with no documentation, re jobs