6 ms·
Kinda funny but I think LLM-assisted workflows are frequently slow -- that is, if I use the "refactor" features in my IDE it is done in a second, if I ask the
by PaulHoule 1y ago
Kinda funny but I think LLM-assisted workflows are frequently slow -- that is, if I use the "refactor" features in my IDE it is done in a second, if I ask the faster kind of assistant it comes back in 30 seconds, if I ask the "agentic" kind of assistant it comes back in 15 minutes.
I asked an agent to write an http endpoint at the end of the work day when I had just 30 min left -- my first thought was "it took 10 minutes to do what would have taken a day", but then I thought, "maybe it was 20 minutes for 4 hours worth of work". The next day I looked at it and found the logic was convoluted, it tried to write good error handling but didn't succeed. I went back and forth and ultimately wound up recoding a lot of stuff manually. In 5 hours I had it done for real, certainly with a better test suite than I would have written on my own and probably better error handling.
See https://www.reddit.com/r/programming/comments/1lxh8ip/study_finds_that_ai_tools_make_experienced/ https://www.reddit.com/r/programming/comments/1lxh8ip/study_...
- citizenpaul 1y agoThe only thing I've found that LLM speeds up my work is a sort of advanced find replace. A prompt like " I want to make this change in the code where any logic deals with XXX. To be/do XXX instead/additionally/somelogicchange/whatever" It has been pretty decent at these types of changes and saves time of poking though and finding all the places I would have updated manually in a way that find/replace never could. Though I've never tried this on a huge code base.
- skydhash 1y agoI supposed you haven’t tried emacs grep mode or vim quickfix? If the change is mechanical, you create a macro and be done in seconds. If it’s not, you still got the high level overview and quick navigation.
- kfajdsl 1y agoFinding and jumping to all the places is usually easy, but non trivial changes often require some understanding of the code beyond just line based regex replace. I could probably spend some time recording a macro that handles all the edge cases, or use some kind of AST based search and replace, but cursor agent does it just fine in the background.
- skydhash 1y agoCode structure is simple. Semantics is where it get tough. So if you have a good understanding of the code (and even when you don't), the overview you get from one of those tools (and the added interactivity) is nice for confirming (understanding) the needed actions that needs to be done. > cursor agent does it just fine in the background That's for a very broad definition of fine. And you still need to review the diff and check the surrounding context of each chunk. I don't see the improvement in metrics like productivity and cognitive load. Especially if you need to do serveral rounds.
- kfajdsl 1y agoYou mentioned grep-mode, which to my knowledge is just bringing up a buffer with all the matches for a regex and easily jumping to each point (I use rg.el myself). For the record, this is basically the same thing as VSCode's search tool. Now, once you have that, to actually make edits, you have to record a macro to apply at each point or just manually do the edit yourself, no? I don't pretend LLMs are perfect, but I certainly think using one is a much better experience for this kind of refactoring than those two options.
- skydhash 1y agoMaybe it's my personal workflow, but I either have sweeping changes (variable names, removing dependencies) which are easily macroable, or very targeted one (extracting functions, decoupling stuff,..,). For both, this navigation is a superpower and coupled with the other tools of emacs/vim, edit is very fast. That rely on a very good mental model of the code, but any question can be answered quickly with the above tools. For me, it's like having a moodboard with code listings.
- Karrot_Kream 1y agoYes I've done this kind of refactoring for ages using emacs macros and grep. Language Server and tree-sitter in emacs has made this faster (when I can get all the dependencies setup correctly that is.) Variable name edits and function extraction is pretty much table stakes in most modern editors like IntelliJ, VSCode, Zed, etc. IIRC Eclipse had this capability 15-20 years ago. I used to have more patience for doing it the grep/macro way in emacs. It used to feel a bit zen, like going through the code and changing all the call-sites to use my new refactor or something. But I've been coding for too long to feel this zen any longer, and my own expectations for output have gotten higher with tools like language-server and tree-sitter. The kind of refactorings I turn to an LLM for are different, like creating interfaces/traits out of structs or joining two different modules together.
- Karrot_Kream 1y agoemacs macros aren't the same. You need to look at the file, observe a pattern, then start recording the macro and hope the pattern holds. An LLM can just do this.
- skydhash 1y agoAnd that's why I mentionned grep-mode, and such other tools. Here is some videos about what I'm talking about https://youtu.be/f2mQXNnChwc?t=2135 https://youtu.be/f2mQXNnChwc?t=2135 https://youtu.be/zxS3zXwV0PU https://youtu.be/zxS3zXwV0PU And for Vim https://youtu.be/wOdL2T4hANk https://youtu.be/wOdL2T4hANk Standard search and replace in other tools pales in comparison.
- Karrot_Kream 1y agoI am familiar with grep-mode and have used that and macro recording for years. I've been using emacs for 20 years. grep-mode (these days I use rg) just brings up all the matches which lets me use a macro that I recorded. That's not the same as telling Claude Code to just make the change. Macros aren't table stakes but find-replace across projects is table stakes in pretty much any post-emacs/vim code editor (and both emacs and vimlikes obviously have plenty of support for this.)
- citizenpaul 1y agoI'm decent at that kind of stuff. However thats not really what I'm talking about. For instance today I needed two logic flows. One for data flowing in one direction. Then a basically but not quite reversed version of the same logic for when the data comes back. I was able to write the first version then tell the LLM "Now duplicate this code but invert the logic for data flowing in the opposite direction." I'm simplifying this whole example obviously but that was the basic task I was working on. It was able to spit out in a few seconds what would have taken me probably more than an hour and at least one tedium headache break. I'm not aware of any pre LLM way to do something like that. Or a little while back I was implementing a basic login/auth for a website. I was experimenting with high output token LLM's (i'm not sure that's the technical term) and asked it to make a very comprehensive login handler. I had to stop it somewhere in the triple digits of cases and functions. Perhaps not a great "pro" example of LLM but even though it was a hilariously over complex setup it did give me some ideas I hadn't thought about. I didn't use any of the code though. Its far from the magic LLM sellers want us to believe but it can save time same as various emac/vim tricks can to devs that want to learn them.
- zahlman 1y ago> A prompt like " I want to make this change in the code where any logic deals with XXX. To be/do XXX instead/additionally/somelogicchange/whatever" If I reached a point where I would find this helpful, I would take this as a sign that I have structured the code wrongly.
- baq 1y agoYou would be right about the code but probably wrong about the you. I’ve done such requests to clean up code written over the years by dozens of other people copying patterns around because ship was king… until it wasn’t. (They worked quite well, btw.)
- zahlman 1y agoIf you're generalizing "you" to all developers, perhaps. I know how I'd respond based on personal experience.
- rtpg 1y agosometimes you want a cutpoint for a refactor and only that refactor. And turns out that there is no nice abstraction that is useful beyond that refactor.
- citizenpaul 1y agoI knew someone would make this comment. I almost added a "I'm probably not leet enough to avoid these situations" disclaimer. It seemed a bit pointlessly self deprecating. You don't always get to choose the state of or the way a system you work in/with is designed. In this case I was working in a limited scripting language that I have no choice about. Keep that nose turned up. I'm sure you are leet10xninja. Maybe work on your reading comprehension before you dump on someone though as I already specified that I greatly simplified for comment sake.
- zahlman 1y agoNo slight was intended. I've learned a lot of techniques; I don't consider this a matter of elitism. I have generally been fortunate to have control over the architecture of my projects; when I encounter something like this in someone else's code, I can at least raise my concern. Sometimes the code is that way because doing it the other obvious way would lead to some other inconvenience. Hence "a sign", not proof. It's worth investigating signs.
- tomrod 1y agoI'm consistently seeing personal and shared anecdotes of a 40%-60% speedup on targeted senior work. As much as I like agents, I am not convinced the human using them can sit back and get lazy quite yet!
- michaelsalim 1y agoCurious, what do you count as senior work?
- tomrod 1y agoRoughly: A senior can write, test, deploy, and possibly maintain a scalable microservice or similar sized project without significant hand-holding in a reasonable amount of time. A junior might be able to write a method used by a class but is still learning significant portions and concepts either in the language, workflow orchestration, or infrastructure. A principal knows how each microservice fits into the larger domain they service, whether they understand all services and all domains they serve. A staff has significant principal understanding across many or all domains an organization uses, builds, and maintains. AI code assistance help increase breadth and, with oversight, improve depth. One can move from the "T" shape to "V" shape skillset far easier, but one must never fully trust AI code assistants.
- stavros 1y agoEeeh, I spend less time writing code, but way more time reviewing and correcting it. I'm not sure I come ahead overall, but it does make development less boilerplaty and more high level, which leads to code that otherwise wouldn't have been written.
- tomrod 1y agoI wonder if you observe this when you use it in a domain you know well versus a domain you know less well. I think LLM assistants help you become functional across a more broad context -- and completely agree that testing and reviewing becomes much, much more important. E.g - a front end dev optimizing database queries, but also being given nonsensical query parameters that don't exist.
- Karrot_Kream 1y agoI guess it depends? The "refactor" stuff, if your IDE or language server can handle it, then yeah I find the LLM slower for sure. But there are other cases than an LLM helps a lot. I was writing some URL canonicalization logic yesterday. Because we rolled this out as an MVP, customers put URLs in all sorts of ways and we stored it into the DB. My initial pass at the logic failed on some cases. Luckily URL canonicalization is pretty trivially testable. So I took the most used customers from our DB, send them to Claude and told Claude to come up with the "minimum spanning test cases" that cover this behavior. This took maybe 5-10 sec. I then told Zed's agent mode using Opus to make me a test file and use these test cases to call my function. I audited the test cases and ended up removing some silly ones. I iterated on my logic and that was that. Definitely faster than having to do this myself.
- cycomanic 1y agoI've already written about this several times here. I think the current trend of LLMs chasing benchmark scores are going in the wrong direction at least as programming tools. In my experience they get it wrong with enough probability, so I always need to check the work. So I end up in a back and forth with the LLM and because of the slow responses it becomes a really painful process and I could often have done the task faster if I sat down and thought about it. What I want is an agent that responds immediately (and I mean in subseconds) even if some benchmark score is 60% instead of 80%.
- pron 1y agoProgrammers (and I'm including myself here) often go to great lengths to not think, to the point of working (with or without a coding assistant) for hours in the hope of avoiding one hour of thinking. What's the saying? "An hour of debugging/programming can save you minutes of thinking," or something like that. In the end, we usually find that we need to do the thinking after all. I think coding assistants would end up being more helpful if, instead of trying to do what they're asked, they would come back with questions that help us (or force us) to think. I wonder if a context prompt that says, "when I ask you to do something, assume I haven't thought the problem through, and before doing anything, ask me leading questions," would help. I think Leslie Lamport once said that the biggest resistance to using TLA+ - a language that helps you, and forces you to think - is because that's the last thing programmers want to do.
- PaulHoule 1y agoSometimes thinking and experimenting go together. I had to do some maintenance on some Typescript/yum that I didn't write but had done a little maintenance. Typescript can make astonishingly complex error messages when types don't match up so I went through a couple of rounds of showing the errors to the assistant and getting suggestions to fix it that were wrong but I got some ideas and did more experiments and over the course of two days (making desired changes along the way) I figured out what was going wrong and cleared up the use of types such that I was really happy with my code and when I saw a red squiggle I usually knew right away what was wrong and if I did ask the assistant it would also get it right right away. I think there's no way I would have understood what was going on without experimenting.
- cornfieldlabs 1y agoI switch to vs code from cursor many times a day just to use their python refactoring feature. The pylance server that comes with cursor doesn't support refactoring.
- resonious 1y agoAs a counter example (re: agents), I routinely delegate simple tasks to Claude Code and get near-perfect results. But I've also had experiences like yours where I ended up wasting more time than saved. I just kept trying with different types of tasks, and narrowed it down to the point where I have a good intuition for what works and what doesn't. The benefit is I can fire off a request on my phone, stick it in my pocket, then do a code review some time later. This process is very low mental overhead for me, so it's a big productivity win.
- xyzzy123 1y agoThats cool, how are you integrating your phone with your Claude workflow?
- ChadNauseam 1y agoI don't know how to do it with Claude Code, but I was at a beach vacation for the past few days and I was studying french on my phone with an webapp that I made. Sometimes I'd notice something bug me, and I used cursor's "background agents" tool to ask it to make a change. This is essentially just a website where you can type in your request, and they allocate a VM, check out your repository, then run the cursor LLM agent inside that VM to implement your requested changes, then push it and create a pull request to your repo. Because I have CI/CD setup, I then just merged the change and waited for it to deploy (usually going for a swim in-between). I realized as I was doing it that I wouldn't be able to tell anyone about it because I would sound like the most obnoxious AI bro ever. But it worked! (For the simple requests I used it on.) The most annoying part was that I had to tell it to run rustfmt every time, because otherwise it would fail CI and I wouldn't be able to merge it. And then it would take forever to install a rust toolchain and figure out how to run clippy and stuff. But it did feel crazy to be able to work on it from the beach. Anyway, I'm apparently not very good at taking vacations, lol
- seren1 1y agoIs this Terragon's offering, or does Cursor also have web-based background agents?
- roncesvalles 1y agoAll the references to LLMs in the article seemed out-of-place like poorly done product placement. LLMs are the anti-thesis of fast. In fact, being slow is a perceived virtue with LLM output. Some sites like Google and Quora (until recently) simulate the slow typed output effect for their pre-cached LLM answers, just for credibility.
- pjmlp 1y agoNot only that, I am already typing enough for coding, I don't want to type on chat windows as well, and so far the voice assistance is so so.