12 ms·

I'm dialing back my LLM usage

422 points by sagacity 1y ago

deleted 1y ago
[deleted]
Nedomas 1y ago
two weeks ago I started heavily using Codex (I have 20y+ dev xp). At first I was very enthusiastic and thought Codex is helping me multiplex myself. But you actually spend so much time trying to explain Codex the most obvious things and it gets them wrong all the time in some kind of nuanced way that in the end you spend more time doing things via Codex than by hand. So I also dialed back Codex usage and got back to doing many more things by hand again because its just so much faster and much more predictable time-wise.
- nsingh2 1y ago
  Same experience, these "background agents" are powered by models that aren't yet capable enough to handle large, tangled or legacy codebases without human guidance. So the background part ends up being functionally useless in my experience.
seanw444 1y ago
This is pretty much the conclusion I've come to as well. It's not good at being an autocomplete for entire chunks of your codebase. You lose the mental model of what is doing what, and exactly where. I prefer to use it as a personalized, faster-iterating StackOverflow. I'll ask it to give me a rundown of a concept I'm not familiar with, or for a general direction to point me in if I'm uncertain of what a good solution would be. Then I'll make the decision, and implement it myself. That workflow has worked out much better for me so far.
- solomonb 1y ago
  I use it the same way but cursor is constantly insisting on making code changes. Is there a trick to get it to introspect on the codebase without wanting to modify it?
  - haiku2077 1y ago
    In Zed you can toggle between read-only and write modes at any point when using the agent. You can also create custom modes that allow the use of specific tools, editing only specific files, etc. Does cursor have a similar feature?
  - nsingh2 1y ago
    In Cursor, there is an 'ask' mode that isn't as over-eager to make edits as the default agent mode.
  - furyofantares 1y ago
    I say something like "without making any code changes right now, investigate blah blah blah" and if I want more than just info "and propose a direction that I can look at", or sometimes give it a file to write a proposal into.
  - deleted 1y ago
    [deleted]
- rane 1y ago
  I don't have this experience. The mental mode might not be quite as strong, but if you always review the code given carefully, you will have a good pretty idea what is where and how things interact.
  - seanw444 1y ago
    It seems to be analogous to just reading a concept, versus reading a concept and taking notes on it to solidify it further. Writing the code myself helps it all solidify in my brain.
travisgriggs 1y ago
This parrots much of my own experience. I don’t have it write of my Python firmware or Elixir backend stuff. What I do let it rough in is web front end stuff. I view the need for and utility of LLMs in the html/css/tailwind/js space as an indictment of complexity and inconsistency. It’s amazing that the web front end stuff has just evolved over the years, organically morphing from one thing to another, but a sound well engineered simple-is-best set of software it is not. And in a world where my efforts will probably work in most browser contexts, no surprise that I’m willing to mix in a tool that will make results that will probably work. A mess is still a mess.
quaintdev 1y ago
LLMs have limits. They are super powerful but they can't make the kind of leap humans can. For example, I asked both Claude and Gemini below problem. "I want to run webserver on Android but it does not allow binding on ports lower than 1000. What are my options?" Both responded with below solutions 1. Use reverse proxy 2. Root the phone 3. Run on higher port Even after asking them to rethink they couldn't come up with the solution I was expecting. The solution to this problem is HTTPS RR records[1]. Both models knew about HTTPS RR but couldn't suggest it as a solution. It's only after I included it in their context both agreed it as a possible solution. [1]: https://rohanrd.xyz/posts/hosting-website-on-phone/ https://rohanrd.xyz/posts/hosting-website-on-phone/
- _flux 1y ago
  TIL. I knew about the SRV reconds—which almost nobody uses I think?—but this was news to me. I guess it's also actually supported, unlike SRV that are more like supported only by some applications? Matrix migrated from SRV to .well-known files for providing the data. (Or I maybe it supports both.)
  - bravetraveler 1y ago
    You'd be surprised at how many games use SRV records. Children struggle with names; let alone ports and modifier keys. At least... this was before multiplayer discovery was commandeered. Matchmaking and so on largely put an end to opportunities.
  - remram 1y ago
    See also SVCB
  - Arathorn 1y ago
    Matrix supports both; when you're trying to get Matrix deployed on someone's domain it's a crapshoot on whether they have permission to write to .well-known on the webroot (and if they do, the chances of it getting vaped by a CMS update are high)... or whether they have permission to set DNS records.
- 1y ago
furyofantares 1y ago
I've gone the other way and put a lot of effort into figuring out how to best utilize these things. It's a rough learning curve and not trivial, especially given how effortless stuff looks and feels at first.
MrGilbert 1y ago
My point of view: LLMs should be taken as a tool, not as a source of wisdom. I know someone who likes to answer people-related questions through a LLM. (E.g.: "What should this person do?" "What should we know about you?" etc.) More than once, this leads to him getting into a state of limbo when he tries to explain what he means with what he wrote. It feels a bit wild - a bit like back in school, when the guy who copied your homework, is forced to explain how he ended up with the solution.
atonse 1y ago
These seem like good checkpoints (and valid criticisms) on the road to progress. But it's also not crazy to think that with LLMs getting smarter (and considerable resources put into making them better at coding), that future versions would clean up and refactor code written by past versions. Correct?
- bwfan123 1y ago
  nope, there are limits to what next-token predictions can do, we we have hit those limits. cursor and the like are great for some usecases - for example a semantic search for relevant code snippets, and autocomplete. But beyond that, they only bring frustration in my use.
  - bunderbunder 1y ago
    Arguably most of the recent improvement in AI coding agents didn't exactly come from getting better at next token prediction in the first place. It came from getting better at context management, and RAG, and improvements on the usable context window size that let you do more with context management and RAG. And I don't really see any reason to declare we've hit the limit of what can be done with those kinds of techniques.
    - bwfan123 1y ago
      I am sure they will continue to improve just as the static-analyzers and linters are improving. But, fundamentally, LLMs lack a theory of the program as intended in this comment https://news.ycombinator.com/item?id=44443109#44444904 https://news.ycombinator.com/item?id=44443109#44444904 . Hence, they can never reach the promised land that is being talked about - unless there are innovations beyond next-token prediction.
      - bunderbunder 1y ago
        They do lack a theory of program. But also, if there's one consistent theme that you can trace through my 25 of studying and working in ML/AI/whateveryouwanttocallit, it's that symbolic reasoning isn't nearly as critical to building useful tools as we like to think it is. In other words, I would be wrong of me to assume that the only way I can think of to go about solving a problem is the only way to do it.
alexvitkov 1y ago
I've found the Cursor autocomplete to be nice, but I've learned to only accept a completion if it's byte for byte what I would've written. With the context of surrounding code it guesses that often enough to be worth the money for me. The chatbot portion of the software is useless.
- cornfieldlabs 1y ago
  For me it's the opposite. Autocomplete suggests the lines I just deleted and also suggests completely useless stuff. I have a shortcut to snooze (it's possible!) it. It interrupts flow my flow a lot. I would rather those stuff myself. Chat mode on the other hand follows my rules really well. I mostly use o3 - it seems to be the only model that has "common sense" in my experience
x187463 1y ago
Am I spending too much time on HN or is every post/comment section filled with this same narrative? Basically, LLMs are exciting but they produce messy code for which the dev feels no ownership. Managing a codebase written by an LLM is difficult because you have not cognitively loaded the entire thing into your head as you do with code written yourself. They're okay for one-off scripts or projects you do not intend to maintain. This is blog post/comment section summary encountered many times per day. The other side of it is people who seem to have 'gotten it' and can dispatch multiple agents to plan/execute/merge changes across a project and want to tell you how awesome their workflow is without actually showing any code.
- fridder 1y ago
  Trying to wade through the hype or doom is bit of a challenge
- noodletheworld 1y ago
  > want to tell you how awesome their workflow is Or, often, sell you something.
  - dkubb 1y ago
    A lot of the time they are selling themselves as influencers on the subject. It’s often a way to get views or attention that they can use in the future.
- totalperspectiv 1y ago
  I think you hit the nail on the head with the mental model part. I really like this method of thinking about programming "Programming as Theory Building" https://gist.github.com/onlurking/fc5c81d18cfce9ff81bc968a7f342fb1#programming-and-the-programmers-knowledge https://gist.github.com/onlurking/fc5c81d18cfce9ff81bc968a7f... I don't mind when other programmers use AI, and use it myself. What I mind is the abdication of responsibility for the code or result. I don't think that we should be issuing a disclaimer when we use AI any more than when I used grep to do the log search. If we use it, we own the result of it as a tool and need to treat it as such. Extra important for generated code.
chasing 1y ago
LLMs save me a lot of time as a software engineer because they save me a ton of time doing either boilerplate work or mundane tasks that are relatively conceptually easy but annoying to actually have to do/type/whatever in an IDE. But I still more-or-less have to think like a software engineer. That's not going to go away. I have to make sure the code remains clean and well-organized -- which, for example, LLMs can help with, but I have to make precision requests and (most importantly) know specifically what I mean by "clean and well-organized." And I always read through and review any generated code and often tweak the output because at the end of the day I am responsible for the code base and I need to verify quality and I need to be able to answer questions and do all of the usual soft-skill engineering stuff. Etc. Etc. So do whatever fits your need. I think LLMs are a massive multiplier because I can focus on the actual engineering stuff and automate away a bunch of the boring shit. But when I read stuff like: "I lost all my trust in LLMs, so I wouldn't give them a big feature again. I'll do very small things like refactoring or a very small-scoped feature." I feel like I'm hearing something like, "I decided to build a house! So I hired some house builders and told them to build me a house with three bedrooms and two bathrooms and they wound up building something that was not at all what I wanted! Why didn't they know I really liked high ceilings?"
- deleted 1y ago
  [deleted]
- patrickmay 1y ago
  > [LLMs] save me a ton of time doing either boilerplate work I hear this frequently from LLM aficionados. I have a couple of questions about it: 1) If there is so much boilerplate that it takes a significant amount of coding time, why haven't you invested in abstracting it away? 2) The time spent actually writing code is not typically the bottleneck in implementing a system. How much do you really save over the development lifecycle when you have to review the LLM output in any case?
  - hedgehog 1y ago
    I don't know about the boilerplate part but when you are e.g. adding a new abstraction that will help simplify an existing pattern across the code base something like Copilot saves a ton of time. Write down what has to happen and why, then let the machine walk across the code base and make updates, update tests and docs, fix whatever ancillary breaks happen, etc. The real payoff is making it cheaper to do exploratory refactors and simple features so you can focus on making the code and overall design better.
ramon156 1y ago
I like Zed's way of doing stuff (Ask mode). Just ask it a question and let it go through the whole thing. I still haven't figured out how to form the question so it doesn't just rail off and start implementing code. I don't care about code, I ask it to either validate my mental model or improve it
- xattt 1y ago
  This is it. This is a new paradigm and a lot of people seem to think that it’s authoritative. It’s decision support tool, and the output still has to pass an internal litmus test. Whether someone’s litmus test is well-developed is another matter.
jjice 1y ago
My favorite use case for LLMs in long term software production (they're pretty great at one off stuff since I don't need to maintain) is as an advanced boiler plate generator. Stuff that can't just be abstracted to a function or class but also require no real thought. Tests are often (depending on what they're testing) in this realm. I was resistant at first, but I love it. It's reduced the parts of my job that I dislike doing because of how monotonous they are and replaced them with a new fun thing to do - optimizing prompts that get it done for me much faster. Writing the prompt and reviewing the code is _so_ much faster on tedious simple stuff and it leaves the interesting, though provoking parts of my work for me to do.
noisy_boy 1y ago
I think they are making me more productive in achieving my targets and worse in my ability to program. They are exactly like steroids - bigger muscles fast but tons of side effects and everything collapses the moment you stop. Companies don't care because they are more concerned about getting to their targets fast instead of your health. Another harmful drug for our brain if consumed without moderation. I won't entirely stop using them but I have already started to actively control/focus my usage.
- criddell 1y ago
  Your steroids comparison made me think of Cal Newport's recent blog post[1] where he argues that AI is making us lazy. He quotes some researchers who hooked people up to EEG machines then had them work. The people working without AI assistance incurred more brain "strain" and that's probably a good thing. But even he doesn't think AI shouldn't be used. Go ahead and use it for stuff like email but don't use it for your core work. [1] https://calnewport.com/does-ai-make-us-lazy/ https://calnewport.com/does-ai-make-us-lazy/
  - causal 1y ago
    I haven't read the entire paper, but just looking at the abstract and conclusion it ironically seems...lazy? Like we're going to have some people use ChatGPT 4 times in 4 months and draw conclusions on long-term brain activity based on that? How do you even control for noise in such a study? I'm generally sympathetic to the idea that LLMs can create atrophy in our ability to code or whatever, but I dislike that this clickbaity study gets shared so much.
  - ukFxqnLa2sBSBf6 1y ago
    Of all the things I think AI shouldn’t be used for AI is one of them (unless you’re like completely illiterate). Whenever I get a “project status update” and the section header emojis show up I instantly just want to throw that garage in the trash.
    - Tijdreiziger 1y ago
jimbokun 1y ago
I think LLMs have made a lot of developers forget the lessons in "Simple Made Easy": https://www.youtube.com/watch?v=SxdOUGdseq4 https://www.youtube.com/watch?v=SxdOUGdseq4 LLMs seem to be really good at reproducing the classic Ball of Mud, that can't really be refactored or understood. There's a lot of power in creating simple components that interact with other simple components to produce complex functionality. While each component is easy to understand and debug and predict its performance. The trick is to figure out how to decompose your complex problem into these simple components and their interactions. I suppose once LLMs get really good at that skill, will be when we really won't need developers any more.
- globular-toast 1y ago
  > I suppose once LLMs get really good at that skill, will be when we really won't need developers any more. I don't really get this argument. So when LLMs become "perfect" software developers are we just going to have them running 24/7 shitting out every conceivable piece of software ever? What would anyone do with that? Or do you expect every doctor, electrician, sales assistant, hairdresser, train driver etc. to start developing their own software on top of their existing job? What's more likely is a few people will make it their jobs to find and break down problems people have that could use a piece of software and develop said piece of software using whatever means they have available to them. Today we call these people software developers.
  - breckenedge 1y ago
    > Or do you expect every doctor, electrician, sales assistant, hairdresser, train driver etc. to start developing their own software on top of their existing job? I started my software career by automating my job, then automating other people’s jobs. Eventually someone decided it would be easier to just hire me as a software engineer. I just met with an architect for adding a deck onto my house (need plans for code compliance). He said he was using AI to write programs that he could use with design software. He demoed how he was using AI to convert his static renders into walkthrough movies.
- kraftman 1y ago
kadhirvelm 1y ago
I wonder if this is as good as LLMs can get, or if this is a transition period between LLM as an assistant, and LLM as a compiler. Where in the latter world we don’t need to care about the code because we just care about the features. We let the LLM deal with the code and we deal with the context, treating code more like a binary. In that world, I’d bet code gets the same treatment as memory management today, where only a small percent of people need to manage it directly and most of us assume it happens correctly enough to not worry about it.
- rzz3 1y ago
  Why wonder if this is “as good as LLMs can get” when we saw such a huge improvement between Claude 3.7 and Claude 4, released what, a couple weeks ago? Of course it isn’t as good as LLMs can get. Give it 3 more weeks and you’ll see it get better again.
  - kadhirvelm 1y ago
    I don’t doubt LLMs will become better assistants over time, as you said every few weeks. I more mean if LLMs will cross the assistant to compiler chasm where we don’t have to think about the code anymore and can focus on just the features
    - kadhirvelm 1y ago
      Wrote more thoughts here: https://resync-games.com/blog/engineering/llms-as-compiler https://resync-games.com/blog/engineering/llms-as-compiler
e12e 1y ago
Previous discussion on the linked blog post: https://news.ycombinator.com/item?id=44003700 https://news.ycombinator.com/item?id=44003700
sarmadgulzar 1y ago
Can relate. I've also shifted towards generating small snippets of code using LLMs, giving them a glance, and asking to write unit tests for them. And then I review the unit tests carefully. But integrating the snippets together into the bigger system, I always do that myself. LLMs can do it sometimes but when it becomes big enough that it can't fit into the context window, then it's a real issue because now LLMs doesn't know what's going on and neither do you. So, I'll advise you to use LLMs to generate tedious bits of code but you must have the overall architecture committed into your memory as well so that when AI messes up, at least you have some clue about how to fix it.
- causal 1y ago
  What's it called when you choose a task because it's easy, even if it's not what you need to do at all? I think that's what LLMs have activated in a lot of us: writing code used to be kinda hard, but now it's super easy, so let's just write more code. The hard parts of engineering have always been decision making, socializing, and validating ideas against cold hard reality. But writing code just got easier so let's do that instead. Prior to LLMs writing 10 lines of code might have been a really productive day, especially if we were able to thoughtfully avoid writing 1,000 unnecessary lines. LLMs do not change this.
  - Velorivox 1y ago
    I'm not sure if there's a name for that specifically, but it seems strongly related to the streetlight effect. [0] [0] https://en.wikipedia.org/wiki/Streetlight_effect https://en.wikipedia.org/wiki/Streetlight_effect
    - causal 1y ago
      Yeah very apt
deleted 1y ago
[deleted]
veselin 1y ago
I think that people are just too quick to assume this is amazing, before it is there. Which doesn't mean it won't get there. Somehow if I take the best models and agents, most hard coding benchmarks are at below 50% and even swe bench verified is like at 75 maybe 80%. Not 95. Assuming agents just solve most problems is incorrect, despite it being really good at first prototypes. Also in my experience agents are great to a point and then fall off a cliff. Not gradually. Just the type of errors you get past one point is so diverse, one cannot even explain it.
hedgehog 1y ago
Over the last couple months I've gone from highly skeptical to a regular user (Copilot in my case). Two big things changed: First, I figured out that only some models are good enough to do the tasks I want (Claude Sonnet 3.7 and 4 out of everything I've tested). Second, it takes some infrastructure. I've added around 1000 words of additional instructions telling Copilot how to operate, and that's on top of tests (which you should have anyway) and 3rd party documentation. I haven't tried the fleet-of-agents thing, one VS Code instance is enough and I want to understand the changes in detail. Edit: In concrete terms the workflow is to allow Copilot to make changes, see what's broken, fix those, review the diff against the goal, simplify the changes, etc, and repeat, until the overall task is done. All hands off.
Const-me 1y ago
I’m using ChatGPT (enterprise version paid by my employer) quite a lot lately, and I find it a useful tool. Here’s what I learned over time. Don’t feed many pages of code to AI, it works best for isolated functions or small classes with little dependencies. In 10% of cases when I ask to generate or complete code, the quality of the code is less than ideal but fixable with extra instructions. In 25% of cases, the quality of generated code is bad and remains so even after telling it what’s wrong and how to fix. When it happens, I simply ignore the AI output and do something else reasonable. Apart from writing code, I find it useful at reviewing new code I wrote. Half of the comments are crap and should be ignored. Some others are questionable. However, I remember a few times when the AI identified actual bugs or other important issues in my code, and proposed fixes. Again, don’t copy-paste many pages at once, do it piecewise. For some niche areas (examples are HLSL shaders, or C++ with SIMD intrinsics) the AI is pretty much useless, probably was not enough training data available. Overall, I believe ChatGPT improved my code quality. Not only as a result of reviews, comments, or generated codes, but also my piecewise copy-pasting workflow improved overall architecture by splitting the codebase into classes/functions/modules/interfaces each doing their own thing.
- wombat-man 1y ago
  Yeah the code review potential is big. I just started using AI for this and it's pretty handy. I agree it's good for helping writing smaller bits like functions. I also use it to help me write unit tests which can be kind of tedious otherwise. I do think that the quality of AI assistance has improved a lot in the past year. So if you tried it before, maybe take another crack at it.
xyst 1y ago
Just like garbage sources on search engines or trash stack overflow answers. There’s still plenty of junk to sift through with LLM. LLM will even through irrelevant data points in the output which causes further churn. I feel not much has changed.
turbofreak 1y ago
Is this Zed Shaw’s blog?
- gpm 1y ago
  Nah, it's the people/company behind the Zed editor, who are in part the people who were originally behind the Atom editor. https://zed.dev/team https://zed.dev/team
andy99 1y ago
I'm realizing that LLMs, for coding in particular but also for many other tasks, are a new version of the fad dieting phenomenon. People really want a quick, low effort fix that appeals to the energy conserving lizard brain while still promising all the results. In reality there aren't shortcuts, there's just tradeoffs, and we all realize it eventually.
AsmodiusVI 1y ago
Really appreciated this take, hits close to home. I’ve found LLMs great for speed and scaffolding, but the more I rely on them, the more I notice my problem-solving instincts getting duller. There’s a tradeoff between convenience and understanding, and it’s easy to miss until something breaks. Still bullish on using AI for exploring ideas or clarifying intent, but I’m trying to be more intentional about when I lean in vs. when I slow down and think things through myself.
kamens 1y ago
Personally, I follow the simple rule: "I type every single character myself. The AI/agent/etc offers inspiration." It's an effective balance between embracing what the tech can do (I'm dialing up my usage) and maintaining my personal connection to the code (I'm having fun + keeping things in my head). I wrote about it: https://kamens.com/blog/code-with-ai-the-hard-way https://kamens.com/blog/code-with-ai-the-hard-way
thimkerbell 1y ago
We need an app to rate posts on how clickbaity their titles are, and let you filter on this value.
jjangkke 1y ago
The problem with zed's narrative is that because he failed to use it in productive ways he wants to dial it back altogether but its not clear what he has actually attempted and people dogpiling here reminds me of artists who are hostile to AI tools, it doesn't accurately reflect the true state of the marketplace which actually puts a lot of value on successful LLM/AI tool use especially in the context of software development. If you extrapolate this blog then we shouldn't be having so much success with LLMs, we shouldn't be able to ship product with fewer people, and we should be hiring junior developers. But the truth of the matter is, especially for folks that work on agents focusing on software development is that we can see a huge tidal shift happening in ways similar to artists, photographers, translators and copywriters have experienced. The blog sells the idea that LLM is not productive and needs to be dialed down does not tell the whole story. This does not mean I am saying LLM should be used in all scenarios, there are clearly situations where it might not be desirable, but overall the productivity hinderance narrative I repeatedly see on HN isn't convincing and I suspect is highly biased.
- tequila_shot 1y ago
  so, this is not from a developer called zed, but instead a developer called Alberto. This is stated in the first line in the article.
delusional 1y ago
I wish there was a browser addon that worked like ublock but for LLM talk. Like just take it all, every blog post, every announcement, every discussion and wipe it all away. I just want humanity to deal with some of our actual issues, like fascism, war in Europe and the middle east, the centralization of our lines of production, the unfairness in our economies. Instead we're stuck talking about if the lie machine can fucking code. God.
- lucasluitjes 1y ago
  Ironically if you wanted to build that accurately and quickly, you would probably end up having an LLM classify content as being LLM-related or not. Keyword-based filtering would have many false positives, and training a model takes more time to build.
- pmxi 1y ago
  I’m sure you could build a prototype add on to do this pretty quickly with Claude Code or the like
bgwalter 1y ago
This is still an ad that tries to lure heretics in by agreeing with them. This is the new agile religion. Here are suit-optimized diagrams: https://zed.dev/agentic-engineering https://zed.dev/agentic-engineering "Interwoven relationship between the predictable & unpredictable."
incomingpain 1y ago
Using gemini cli, I really need to try out claude code 1 day, and you ask it to make a change and it gives you the diff on what it plans to change. You can say no, then give it more specific instructions like "keep it more simple" or "you dont need that library to be imported" You can read the code and ensure you understand what it's doing.
specproc 1y ago
I spent today rewriting a cloud function I'd done with the "help" of an LLM. Looked like dog shit, but worked fine till it hit some edge cases. Had to break the whole thing down again and pretty much start from scratch. Ultimately not a bad day's work, and I still had it on for autocomplete on doc-strings and such, but like fuck will I be letting an agent near code I do for money again in the near future.
obirunda 1y ago
The dichotomy between the people who are "orchestrating" agents to build software and the people experiencing this less than ideal outcomes from LLMs is fascinating. I don't think LLM for coding productivity is all hype but I think for the people who "see the magic" there are many illusions here similar to those who fall prey to an MLM pitch. You can see all the claims aren't necessarily unfounded, but the lack of guaranteed reproducibility leaves the door open for many caveats in favor of belief for the believer and cynicism for everybody else. For the believers if it's not working for one person, it's a skill issue related to providing the best prompt, the right rules, the perfect context and so forth. At what point is this a roundabout way of doing it yourself anyway?
stephendause 1y ago
One point I haven't seen made elsewhere yet is that LLMs can occasionally make you less productive. If they hallucinate a promising-seeming answer and send you down a path that you wouldn't have gone down otherwise, they can really waste your time. I think on net, they are helpful, especially if you check their sources (which might not always back up what they are saying!). But it's good to keep in mind that sometimes doing it yourself is actually faster.
- Anamon 1y ago
  That was my main reason for having dropped LLM coding assistance completely for now. It wasted so much of my time and energy. I got the occasional helpful response, but not before four that were almost correct, which of course is a much bigger time sink than when it's obviously wrong. Another fifteen minutes spent investigating a solution that turns out to be bogus. I got the usual gaslighting that I'm just using the wrong models, or using them wrong, etc. But when I watched others, I found that their results were just as iffy, they just usually didn't bother to check them once they looked plausible.
KaiMagnus 1y ago
I’ve been starting my prompts more and more with the phrase „Let’s brainstorm“. Really powerful seeing different options, especially based on your codebase. > I wouldn't give them a big feature again. I'll do very small things like refactoring or a very small-scoped feature. That really resonates with me. Anything larger often ends badly and I can feel the „tech debt“ building in my head with each minute Copilot is running. I do like the feeling though when you understood a problem already, write a detailed prompt to nudge the AI into the right direction, and it executes just like you wanted. After all, problem solving is why I’m here and writing code is just the vehicle for it.
i_love_retros 1y ago
Uh oh, I think the bubble is bursting. Personally the initial excitement has worn off for me and I am enjoying writing code myself and just using kagi assistant to ask the odd question, mostly research. When a team mate who bangs on about how we should all be using ai tried to demo it and got things in a bit of a mess, I knew we had peaked. And all that money invested into the hype!
href 1y ago
I've dialed down a lot as well. The answers I got for my queries were too often plain wrong. I instead started asking where I might look something up - in what man page, or in which documentation. Then I go read that. This helps me build a better mental map about where information is found (e.g., in what man page), decreasing both my reliance on search engines, and LLMs in the long run. LLMs have their uses, but they are just a tool, and an imprecise one at that.
dwoldrich 1y ago
I personally believe it's a mistake to invite AI into your editor/IDE. Keep it separate to the browser, keep discrete, concise question and answer threads. Copy and paste whenever it delivers some gold (that comes with all the copy-pasta dangers, I know - oh, don't I know it!) It's important to always maintain the developer role, don't ever surrender it.
somewhereoutth 1y ago
Unfortunately it seems that nobody 'dialed back' LLM usage for the summary on that page - a good example of how un-human such text can feel to read.
65 1y ago
LLMs require fuzzy input and are thus good for fuzzy output, mostly things like recommendations and options. I just do not see a scenario where fuzzy input can lead to absolute, opinionated output unless extremely simple and mostly done before already. Programming, design, writing, etc. all require opinions and an absolute output from the author to be quality.
Gepsens 1y ago
Llms are not a magic wand you can wave at anything and get your work cut out for you. What's new ?
piker 1y ago
Credit to the Zed team here for publishing something somewhat against its book.
nvahalik 1y ago
Sounds like he shot for the moon and missed. I've been allowing LLMs to do more "background" work for me. Giving me some room to experiment with stuff so that I can come back in 10-15 minutes and see what it's done. The key things I've come to are that it HAS to be fairly limited. Giving it a big task like refactoring a code base won't work. Giving it an example can help dramatically. If you haven't "trained" it by giving it context or adding your CLAUDE.md file, you'll end up finding it doing things you don't want it to do. Another great task I've been giving it while I'm working on other things is generating docs for existing features and modules. It is surprisingly good at looking at events and following those events to see where they go and generating diagrams and he like.
cadamsdotcom 1y ago
LLMs give you a power tool after you spent your whole career using hand tools. A chainsaw and chisel do different things and are made for different situations. It’s great to have chainsaws, no longer must we chop down a giant tree with a chisel. On the other hand there’s plenty of room in the trade for handcraft. You still need to use that chisel to smooth off the fine edges of your chainsaw work, so your teammates don’t get splinters.
jemiluv8 1y ago
When I first came across the idea of vibe coding, my first reaction was that this was taking things too far. Isn't it enough that your LLM can help you do - autocomplete - suggest possible solutions to a problem you've taken the time to understand - helps you spend less time reading documentation and instead help guide your approach and sometimes even helps you identify obscure apis that could help you get shit done - help you review your code - come up with multiple designs for a solutions - evaluate multiple designs you come up with for trade-offs - help you understand your problem better and the available apis - write a prototype of some piece of code I feel like LLMs are already doing quite a lot. I spend less time rummaging through documentation or trying to remember obscure api's or other pieces of code in a software project. All I need is a strong mental model about the project and how things are done. There is a lot of obvious heavy lifting that LLMs are doing that I for one am not able to take for granted. For people facing constraints similar to those in a resource constrained economic environment, the benefits of any technology that helps them spend less time doing work that doesn't deliver value is immediately visible/obvious/apparent. It is no longer an argument about whether it is a hype or something, it is more about how best to use it to achieve your goals. Forget the hype. Forget the marketing of AI companies - they have to do that to sell their products - nothing wrong with that. Don't let companies or bloggers set your own expectations of what could or should be done with this piece of tech. Just get on the bandwagon and experiment and find out what is too much. In the end I feel we will all come from these experiments knowing that LLMs are already doing quite a lot. TRIVIA I even came by this article https://www.greptile.com/blog/ai-code-reviews-conflict https://www.greptile.com/blog/ai-code-reviews-conflict. That clearly pointed out how LLM reliance can bring both the 10x dev and 1x dev closer to a median of "goodness". So the 10x dev is probably worse and the 1x dev ends up getting better - I'm probably that guy because I tend to mis subtle things in code and copilot review has had my ass for a while now - I haven't had defects like that in a while.
smoody07 1y ago
"Why I'm dialing back my high level language usage"
ivraatiems 1y ago
The way that LLMs are used/are encouraged by business right now is evidence that they are mostly being pushed by people who don't understand software. I don't know very many actual software engineers who advocate for vibe-coding or using LLMs this way. I know ton of engineers who advocate using them as helpful tools, myself included (in fact I changed my opinion on it as their capabilities grew, and I'll continue to do so). Every tool is just a tool. No tool is a solution. Until and unless we hit AGI, only the human brain is that.
halis 1y ago
It’s almost like the LLMs are simply a glorified autocomplete and have no actual understanding of anything they’re doing. Huh weird!
eviks 1y ago
> Alberto initially embraced LLMs with genuine enthusiasm, hoping they would revolutionize his development workflow. Is there any other way but down from such revolutionary ungrounded expectations?
matrik 1y ago
I find LLMs very efficient in a lot of things except writing big chunks of code that require some organization. They’re good at closing knowlwdge gaps, finding strategies to solve problems, writing code for defined scopes and even reviewing large PRs. However, as others said they can easily alienate you from your own project.
Amaury-El 1y ago
[dead]
careful_ai 1y ago
Too many of us fall into “prompt autopilot” mode—reaching for AI before we think. Your post calls it out beautifully: step back, reclaim the muscle memory of creative problem solving. LLMs should supplement, not substitute. That discipline often separates thoughtful integration from dependency.
deterministic 1y ago
Using AI definitely makes me more productive. However I spend 80% of my "AI time" fixing mistakes made by the AI or explaining why the solution doesn't work.
billbrown 1y ago
The use case for LLM assistance that provides value for me is solving obscure lint or static analysis warnings and errors. I take the message, provide the surrounding code, and it gives me a few approaches to solve them. More than half the time, the resolution is there and I can copy the relevant bit in the literal verbiage. (The other times it's garbage but at least I can see that this is going to require some AI—Actual Intelligence.)

422 points by sagacity 1y ago

deleted 1y ago

[deleted]

two weeks ago I started heavily using Codex (I have 20y+ dev xp). At first I was very enthusiastic and thought Codex is helping me multiplex myself. But you actually spend so much time trying to explain Codex the most obvious things and it gets them wrong all the time in some kind of nuanced way that in the end you spend more time doing things via Codex than by hand. So I also dialed back Codex usage and got back to doing many more things by hand again because its just so much faster and much more predictable time-wise.

nsingh2 1y ago
Same experience, these "background agents" are powered by models that aren't yet capable enough to handle large, tangled or legacy codebases without human guidance. So the background part ends up being functionally useless in my experience.

seanw444 1y ago

This is pretty much the conclusion I've come to as well. It's not good at being an autocomplete for entire chunks of your codebase. You lose the mental model of what is doing what, and exactly where. I prefer to use it as a personalized, faster-iterating StackOverflow. I'll ask it to give me a rundown of a concept I'm not familiar with, or for a general direction to point me in if I'm uncertain of what a good solution would be. Then I'll make the decision, and implement it myself. That workflow has worked out much better for me so far.

solomonb 1y ago
I use it the same way but cursor is constantly insisting on making code changes. Is there a trick to get it to introspect on the codebase without wanting to modify it?
- haiku2077 1y ago
  In Zed you can toggle between read-only and write modes at any point when using the agent. You can also create custom modes that allow the use of specific tools, editing only specific files, etc. Does cursor have a similar feature?
- nsingh2 1y ago
  In Cursor, there is an 'ask' mode that isn't as over-eager to make edits as the default agent mode.
- furyofantares 1y ago
  I say something like "without making any code changes right now, investigate blah blah blah" and if I want more than just info "and propose a direction that I can look at", or sometimes give it a file to write a proposal into.
- deleted 1y ago
  [deleted]
rane 1y ago
I don't have this experience. The mental mode might not be quite as strong, but if you always review the code given carefully, you will have a good pretty idea what is where and how things interact.
- seanw444 1y ago
  It seems to be analogous to just reading a concept, versus reading a concept and taking notes on it to solidify it further. Writing the code myself helps it all solidify in my brain.

travisgriggs 1y ago

This parrots much of my own experience. I don’t have it write of my Python firmware or Elixir backend stuff. What I do let it rough in is web front end stuff. I view the need for and utility of LLMs in the html/css/tailwind/js space as an indictment of complexity and inconsistency. It’s amazing that the web front end stuff has just evolved over the years, organically morphing from one thing to another, but a sound well engineered simple-is-best set of software it is not. And in a world where my efforts will probably work in most browser contexts, no surprise that I’m willing to mix in a tool that will make results that will probably work. A mess is still a mess.

quaintdev 1y ago

LLMs have limits. They are super powerful but they can't make the kind of leap humans can. For example, I asked both Claude and Gemini below problem. "I want to run webserver on Android but it does not allow binding on ports lower than 1000. What are my options?" Both responded with below solutions 1. Use reverse proxy 2. Root the phone 3. Run on higher port Even after asking them to rethink they couldn't come up with the solution I was expecting. The solution to this problem is HTTPS RR records[1]. Both models knew about HTTPS RR but couldn't suggest it as a solution. It's only after I included it in their context both agreed it as a possible solution. [1]: https://rohanrd.xyz/posts/hosting-website-on-phone/ https://rohanrd.xyz/posts/hosting-website-on-phone/

_flux 1y ago
TIL. I knew about the SRV reconds—which almost nobody uses I think?—but this was news to me. I guess it's also actually supported, unlike SRV that are more like supported only by some applications? Matrix migrated from SRV to .well-known files for providing the data. (Or I maybe it supports both.)
- bravetraveler 1y ago
  You'd be surprised at how many games use SRV records. Children struggle with names; let alone ports and modifier keys. At least... this was before multiplayer discovery was commandeered. Matchmaking and so on largely put an end to opportunities.
- remram 1y ago
  See also SVCB
- Arathorn 1y ago
  Matrix supports both; when you're trying to get Matrix deployed on someone's domain it's a crapshoot on whether they have permission to write to .well-known on the webroot (and if they do, the chances of it getting vaped by a CMS update are high)... or whether they have permission to set DNS records.
1y ago

furyofantares 1y ago

I've gone the other way and put a lot of effort into figuring out how to best utilize these things. It's a rough learning curve and not trivial, especially given how effortless stuff looks and feels at first.

MrGilbert 1y ago

My point of view: LLMs should be taken as a tool, not as a source of wisdom. I know someone who likes to answer people-related questions through a LLM. (E.g.: "What should this person do?" "What should we know about you?" etc.) More than once, this leads to him getting into a state of limbo when he tries to explain what he means with what he wrote. It feels a bit wild - a bit like back in school, when the guy who copied your homework, is forced to explain how he ended up with the solution.

atonse 1y ago

These seem like good checkpoints (and valid criticisms) on the road to progress. But it's also not crazy to think that with LLMs getting smarter (and considerable resources put into making them better at coding), that future versions would clean up and refactor code written by past versions. Correct?

bwfan123 1y ago
nope, there are limits to what next-token predictions can do, we we have hit those limits. cursor and the like are great for some usecases - for example a semantic search for relevant code snippets, and autocomplete. But beyond that, they only bring frustration in my use.
- bunderbunder 1y ago
  Arguably most of the recent improvement in AI coding agents didn't exactly come from getting better at next token prediction in the first place. It came from getting better at context management, and RAG, and improvements on the usable context window size that let you do more with context management and RAG. And I don't really see any reason to declare we've hit the limit of what can be done with those kinds of techniques.
  - bwfan123 1y ago
    I am sure they will continue to improve just as the static-analyzers and linters are improving. But, fundamentally, LLMs lack a theory of the program as intended in this comment https://news.ycombinator.com/item?id=44443109#44444904 https://news.ycombinator.com/item?id=44443109#44444904 . Hence, they can never reach the promised land that is being talked about - unless there are innovations beyond next-token prediction.
    - bunderbunder 1y ago
      They do lack a theory of program. But also, if there's one consistent theme that you can trace through my 25 of studying and working in ML/AI/whateveryouwanttocallit, it's that symbolic reasoning isn't nearly as critical to building useful tools as we like to think it is. In other words, I would be wrong of me to assume that the only way I can think of to go about solving a problem is the only way to do it.

alexvitkov 1y ago

I've found the Cursor autocomplete to be nice, but I've learned to only accept a completion if it's byte for byte what I would've written. With the context of surrounding code it guesses that often enough to be worth the money for me. The chatbot portion of the software is useless.

cornfieldlabs 1y ago
For me it's the opposite. Autocomplete suggests the lines I just deleted and also suggests completely useless stuff. I have a shortcut to snooze (it's possible!) it. It interrupts flow my flow a lot. I would rather those stuff myself. Chat mode on the other hand follows my rules really well. I mostly use o3 - it seems to be the only model that has "common sense" in my experience

x187463 1y ago

Am I spending too much time on HN or is every post/comment section filled with this same narrative? Basically, LLMs are exciting but they produce messy code for which the dev feels no ownership. Managing a codebase written by an LLM is difficult because you have not cognitively loaded the entire thing into your head as you do with code written yourself. They're okay for one-off scripts or projects you do not intend to maintain. This is blog post/comment section summary encountered many times per day. The other side of it is people who seem to have 'gotten it' and can dispatch multiple agents to plan/execute/merge changes across a project and want to tell you how awesome their workflow is without actually showing any code.

fridder 1y ago
Trying to wade through the hype or doom is bit of a challenge
noodletheworld 1y ago
> want to tell you how awesome their workflow is Or, often, sell you something.
- dkubb 1y ago
  A lot of the time they are selling themselves as influencers on the subject. It’s often a way to get views or attention that they can use in the future.
totalperspectiv 1y ago
I think you hit the nail on the head with the mental model part. I really like this method of thinking about programming "Programming as Theory Building" https://gist.github.com/onlurking/fc5c81d18cfce9ff81bc968a7f342fb1#programming-and-the-programmers-knowledge https://gist.github.com/onlurking/fc5c81d18cfce9ff81bc968a7f... I don't mind when other programmers use AI, and use it myself. What I mind is the abdication of responsibility for the code or result. I don't think that we should be issuing a disclaimer when we use AI any more than when I used grep to do the log search. If we use it, we own the result of it as a tool and need to treat it as such. Extra important for generated code.

chasing 1y ago

LLMs save me a lot of time as a software engineer because they save me a ton of time doing either boilerplate work or mundane tasks that are relatively conceptually easy but annoying to actually have to do/type/whatever in an IDE. But I still more-or-less have to think like a software engineer. That's not going to go away. I have to make sure the code remains clean and well-organized -- which, for example, LLMs can help with, but I have to make precision requests and (most importantly) know specifically what I mean by "clean and well-organized." And I always read through and review any generated code and often tweak the output because at the end of the day I am responsible for the code base and I need to verify quality and I need to be able to answer questions and do all of the usual soft-skill engineering stuff. Etc. Etc. So do whatever fits your need. I think LLMs are a massive multiplier because I can focus on the actual engineering stuff and automate away a bunch of the boring shit. But when I read stuff like: "I lost all my trust in LLMs, so I wouldn't give them a big feature again. I'll do very small things like refactoring or a very small-scoped feature." I feel like I'm hearing something like, "I decided to build a house! So I hired some house builders and told them to build me a house with three bedrooms and two bathrooms and they wound up building something that was not at all what I wanted! Why didn't they know I really liked high ceilings?"

deleted 1y ago
[deleted]
patrickmay 1y ago
> [LLMs] save me a ton of time doing either boilerplate work I hear this frequently from LLM aficionados. I have a couple of questions about it: 1) If there is so much boilerplate that it takes a significant amount of coding time, why haven't you invested in abstracting it away? 2) The time spent actually writing code is not typically the bottleneck in implementing a system. How much do you really save over the development lifecycle when you have to review the LLM output in any case?
- hedgehog 1y ago
  I don't know about the boilerplate part but when you are e.g. adding a new abstraction that will help simplify an existing pattern across the code base something like Copilot saves a ton of time. Write down what has to happen and why, then let the machine walk across the code base and make updates, update tests and docs, fix whatever ancillary breaks happen, etc. The real payoff is making it cheaper to do exploratory refactors and simple features so you can focus on making the code and overall design better.

ramon156 1y ago

I like Zed's way of doing stuff (Ask mode). Just ask it a question and let it go through the whole thing. I still haven't figured out how to form the question so it doesn't just rail off and start implementing code. I don't care about code, I ask it to either validate my mental model or improve it

xattt 1y ago
This is it. This is a new paradigm and a lot of people seem to think that it’s authoritative. It’s decision support tool, and the output still has to pass an internal litmus test. Whether someone’s litmus test is well-developed is another matter.

jjice 1y ago

My favorite use case for LLMs in long term software production (they're pretty great at one off stuff since I don't need to maintain) is as an advanced boiler plate generator. Stuff that can't just be abstracted to a function or class but also require no real thought. Tests are often (depending on what they're testing) in this realm. I was resistant at first, but I love it. It's reduced the parts of my job that I dislike doing because of how monotonous they are and replaced them with a new fun thing to do - optimizing prompts that get it done for me much faster. Writing the prompt and reviewing the code is _so_ much faster on tedious simple stuff and it leaves the interesting, though provoking parts of my work for me to do.

noisy_boy 1y ago

I think they are making me more productive in achieving my targets and worse in my ability to program. They are exactly like steroids - bigger muscles fast but tons of side effects and everything collapses the moment you stop. Companies don't care because they are more concerned about getting to their targets fast instead of your health. Another harmful drug for our brain if consumed without moderation. I won't entirely stop using them but I have already started to actively control/focus my usage.

criddell 1y ago
Your steroids comparison made me think of Cal Newport's recent blog post[1] where he argues that AI is making us lazy. He quotes some researchers who hooked people up to EEG machines then had them work. The people working without AI assistance incurred more brain "strain" and that's probably a good thing. But even he doesn't think AI shouldn't be used. Go ahead and use it for stuff like email but don't use it for your core work. [1] https://calnewport.com/does-ai-make-us-lazy/ https://calnewport.com/does-ai-make-us-lazy/
- causal 1y ago
  I haven't read the entire paper, but just looking at the abstract and conclusion it ironically seems...lazy? Like we're going to have some people use ChatGPT 4 times in 4 months and draw conclusions on long-term brain activity based on that? How do you even control for noise in such a study? I'm generally sympathetic to the idea that LLMs can create atrophy in our ability to code or whatever, but I dislike that this clickbaity study gets shared so much.
- ukFxqnLa2sBSBf6 1y ago
  Of all the things I think AI shouldn’t be used for AI is one of them (unless you’re like completely illiterate). Whenever I get a “project status update” and the section header emojis show up I instantly just want to throw that garage in the trash.
  - Tijdreiziger 1y ago

jimbokun 1y ago

I think LLMs have made a lot of developers forget the lessons in "Simple Made Easy": https://www.youtube.com/watch?v=SxdOUGdseq4 https://www.youtube.com/watch?v=SxdOUGdseq4 LLMs seem to be really good at reproducing the classic Ball of Mud, that can't really be refactored or understood. There's a lot of power in creating simple components that interact with other simple components to produce complex functionality. While each component is easy to understand and debug and predict its performance. The trick is to figure out how to decompose your complex problem into these simple components and their interactions. I suppose once LLMs get really good at that skill, will be when we really won't need developers any more.

globular-toast 1y ago
> I suppose once LLMs get really good at that skill, will be when we really won't need developers any more. I don't really get this argument. So when LLMs become "perfect" software developers are we just going to have them running 24/7 shitting out every conceivable piece of software ever? What would anyone do with that? Or do you expect every doctor, electrician, sales assistant, hairdresser, train driver etc. to start developing their own software on top of their existing job? What's more likely is a few people will make it their jobs to find and break down problems people have that could use a piece of software and develop said piece of software using whatever means they have available to them. Today we call these people software developers.
- breckenedge 1y ago
  > Or do you expect every doctor, electrician, sales assistant, hairdresser, train driver etc. to start developing their own software on top of their existing job? I started my software career by automating my job, then automating other people’s jobs. Eventually someone decided it would be easier to just hire me as a software engineer. I just met with an architect for adding a deck onto my house (need plans for code compliance). He said he was using AI to write programs that he could use with design software. He demoed how he was using AI to convert his static renders into walkthrough movies.
kraftman 1y ago

kadhirvelm 1y ago

I wonder if this is as good as LLMs can get, or if this is a transition period between LLM as an assistant, and LLM as a compiler. Where in the latter world we don’t need to care about the code because we just care about the features. We let the LLM deal with the code and we deal with the context, treating code more like a binary. In that world, I’d bet code gets the same treatment as memory management today, where only a small percent of people need to manage it directly and most of us assume it happens correctly enough to not worry about it.

rzz3 1y ago
Why wonder if this is “as good as LLMs can get” when we saw such a huge improvement between Claude 3.7 and Claude 4, released what, a couple weeks ago? Of course it isn’t as good as LLMs can get. Give it 3 more weeks and you’ll see it get better again.
- kadhirvelm 1y ago
  I don’t doubt LLMs will become better assistants over time, as you said every few weeks. I more mean if LLMs will cross the assistant to compiler chasm where we don’t have to think about the code anymore and can focus on just the features
  - kadhirvelm 1y ago
    Wrote more thoughts here: https://resync-games.com/blog/engineering/llms-as-compiler https://resync-games.com/blog/engineering/llms-as-compiler

e12e 1y ago

Previous discussion on the linked blog post: https://news.ycombinator.com/item?id=44003700 https://news.ycombinator.com/item?id=44003700

sarmadgulzar 1y ago

Can relate. I've also shifted towards generating small snippets of code using LLMs, giving them a glance, and asking to write unit tests for them. And then I review the unit tests carefully. But integrating the snippets together into the bigger system, I always do that myself. LLMs can do it sometimes but when it becomes big enough that it can't fit into the context window, then it's a real issue because now LLMs doesn't know what's going on and neither do you. So, I'll advise you to use LLMs to generate tedious bits of code but you must have the overall architecture committed into your memory as well so that when AI messes up, at least you have some clue about how to fix it.

causal 1y ago
What's it called when you choose a task because it's easy, even if it's not what you need to do at all? I think that's what LLMs have activated in a lot of us: writing code used to be kinda hard, but now it's super easy, so let's just write more code. The hard parts of engineering have always been decision making, socializing, and validating ideas against cold hard reality. But writing code just got easier so let's do that instead. Prior to LLMs writing 10 lines of code might have been a really productive day, especially if we were able to thoughtfully avoid writing 1,000 unnecessary lines. LLMs do not change this.
- Velorivox 1y ago
  I'm not sure if there's a name for that specifically, but it seems strongly related to the streetlight effect. [0] [0] https://en.wikipedia.org/wiki/Streetlight_effect https://en.wikipedia.org/wiki/Streetlight_effect
  - causal 1y ago
    Yeah very apt

deleted 1y ago

[deleted]

veselin 1y ago

I think that people are just too quick to assume this is amazing, before it is there. Which doesn't mean it won't get there. Somehow if I take the best models and agents, most hard coding benchmarks are at below 50% and even swe bench verified is like at 75 maybe 80%. Not 95. Assuming agents just solve most problems is incorrect, despite it being really good at first prototypes. Also in my experience agents are great to a point and then fall off a cliff. Not gradually. Just the type of errors you get past one point is so diverse, one cannot even explain it.

hedgehog 1y ago

Over the last couple months I've gone from highly skeptical to a regular user (Copilot in my case). Two big things changed: First, I figured out that only some models are good enough to do the tasks I want (Claude Sonnet 3.7 and 4 out of everything I've tested). Second, it takes some infrastructure. I've added around 1000 words of additional instructions telling Copilot how to operate, and that's on top of tests (which you should have anyway) and 3rd party documentation. I haven't tried the fleet-of-agents thing, one VS Code instance is enough and I want to understand the changes in detail. Edit: In concrete terms the workflow is to allow Copilot to make changes, see what's broken, fix those, review the diff against the goal, simplify the changes, etc, and repeat, until the overall task is done. All hands off.

Const-me 1y ago

I’m using ChatGPT (enterprise version paid by my employer) quite a lot lately, and I find it a useful tool. Here’s what I learned over time. Don’t feed many pages of code to AI, it works best for isolated functions or small classes with little dependencies. In 10% of cases when I ask to generate or complete code, the quality of the code is less than ideal but fixable with extra instructions. In 25% of cases, the quality of generated code is bad and remains so even after telling it what’s wrong and how to fix. When it happens, I simply ignore the AI output and do something else reasonable. Apart from writing code, I find it useful at reviewing new code I wrote. Half of the comments are crap and should be ignored. Some others are questionable. However, I remember a few times when the AI identified actual bugs or other important issues in my code, and proposed fixes. Again, don’t copy-paste many pages at once, do it piecewise. For some niche areas (examples are HLSL shaders, or C++ with SIMD intrinsics) the AI is pretty much useless, probably was not enough training data available. Overall, I believe ChatGPT improved my code quality. Not only as a result of reviews, comments, or generated codes, but also my piecewise copy-pasting workflow improved overall architecture by splitting the codebase into classes/functions/modules/interfaces each doing their own thing.

wombat-man 1y ago
Yeah the code review potential is big. I just started using AI for this and it's pretty handy. I agree it's good for helping writing smaller bits like functions. I also use it to help me write unit tests which can be kind of tedious otherwise. I do think that the quality of AI assistance has improved a lot in the past year. So if you tried it before, maybe take another crack at it.

xyst 1y ago

Just like garbage sources on search engines or trash stack overflow answers. There’s still plenty of junk to sift through with LLM. LLM will even through irrelevant data points in the output which causes further churn. I feel not much has changed.

turbofreak 1y ago

Is this Zed Shaw’s blog?

gpm 1y ago
Nah, it's the people/company behind the Zed editor, who are in part the people who were originally behind the Atom editor. https://zed.dev/team https://zed.dev/team

andy99 1y ago

I'm realizing that LLMs, for coding in particular but also for many other tasks, are a new version of the fad dieting phenomenon. People really want a quick, low effort fix that appeals to the energy conserving lizard brain while still promising all the results. In reality there aren't shortcuts, there's just tradeoffs, and we all realize it eventually.

AsmodiusVI 1y ago

Really appreciated this take, hits close to home. I’ve found LLMs great for speed and scaffolding, but the more I rely on them, the more I notice my problem-solving instincts getting duller. There’s a tradeoff between convenience and understanding, and it’s easy to miss until something breaks. Still bullish on using AI for exploring ideas or clarifying intent, but I’m trying to be more intentional about when I lean in vs. when I slow down and think things through myself.

kamens 1y ago

Personally, I follow the simple rule: "I type every single character myself. The AI/agent/etc offers inspiration." It's an effective balance between embracing what the tech can do (I'm dialing up my usage) and maintaining my personal connection to the code (I'm having fun + keeping things in my head). I wrote about it: https://kamens.com/blog/code-with-ai-the-hard-way https://kamens.com/blog/code-with-ai-the-hard-way

thimkerbell 1y ago

We need an app to rate posts on how clickbaity their titles are, and let you filter on this value.

jjangkke 1y ago

The problem with zed's narrative is that because he failed to use it in productive ways he wants to dial it back altogether but its not clear what he has actually attempted and people dogpiling here reminds me of artists who are hostile to AI tools, it doesn't accurately reflect the true state of the marketplace which actually puts a lot of value on successful LLM/AI tool use especially in the context of software development. If you extrapolate this blog then we shouldn't be having so much success with LLMs, we shouldn't be able to ship product with fewer people, and we should be hiring junior developers. But the truth of the matter is, especially for folks that work on agents focusing on software development is that we can see a huge tidal shift happening in ways similar to artists, photographers, translators and copywriters have experienced. The blog sells the idea that LLM is not productive and needs to be dialed down does not tell the whole story. This does not mean I am saying LLM should be used in all scenarios, there are clearly situations where it might not be desirable, but overall the productivity hinderance narrative I repeatedly see on HN isn't convincing and I suspect is highly biased.

tequila_shot 1y ago
so, this is not from a developer called zed, but instead a developer called Alberto. This is stated in the first line in the article.

delusional 1y ago

I wish there was a browser addon that worked like ublock but for LLM talk. Like just take it all, every blog post, every announcement, every discussion and wipe it all away. I just want humanity to deal with some of our actual issues, like fascism, war in Europe and the middle east, the centralization of our lines of production, the unfairness in our economies. Instead we're stuck talking about if the lie machine can fucking code. God.

lucasluitjes 1y ago
Ironically if you wanted to build that accurately and quickly, you would probably end up having an LLM classify content as being LLM-related or not. Keyword-based filtering would have many false positives, and training a model takes more time to build.
pmxi 1y ago
I’m sure you could build a prototype add on to do this pretty quickly with Claude Code or the like

bgwalter 1y ago

This is still an ad that tries to lure heretics in by agreeing with them. This is the new agile religion. Here are suit-optimized diagrams: https://zed.dev/agentic-engineering https://zed.dev/agentic-engineering "Interwoven relationship between the predictable & unpredictable."

incomingpain 1y ago

Using gemini cli, I really need to try out claude code 1 day, and you ask it to make a change and it gives you the diff on what it plans to change. You can say no, then give it more specific instructions like "keep it more simple" or "you dont need that library to be imported" You can read the code and ensure you understand what it's doing.

specproc 1y ago

I spent today rewriting a cloud function I'd done with the "help" of an LLM. Looked like dog shit, but worked fine till it hit some edge cases. Had to break the whole thing down again and pretty much start from scratch. Ultimately not a bad day's work, and I still had it on for autocomplete on doc-strings and such, but like fuck will I be letting an agent near code I do for money again in the near future.

obirunda 1y ago

The dichotomy between the people who are "orchestrating" agents to build software and the people experiencing this less than ideal outcomes from LLMs is fascinating. I don't think LLM for coding productivity is all hype but I think for the people who "see the magic" there are many illusions here similar to those who fall prey to an MLM pitch. You can see all the claims aren't necessarily unfounded, but the lack of guaranteed reproducibility leaves the door open for many caveats in favor of belief for the believer and cynicism for everybody else. For the believers if it's not working for one person, it's a skill issue related to providing the best prompt, the right rules, the perfect context and so forth. At what point is this a roundabout way of doing it yourself anyway?

stephendause 1y ago

One point I haven't seen made elsewhere yet is that LLMs can occasionally make you less productive. If they hallucinate a promising-seeming answer and send you down a path that you wouldn't have gone down otherwise, they can really waste your time. I think on net, they are helpful, especially if you check their sources (which might not always back up what they are saying!). But it's good to keep in mind that sometimes doing it yourself is actually faster.

Anamon 1y ago
That was my main reason for having dropped LLM coding assistance completely for now. It wasted so much of my time and energy. I got the occasional helpful response, but not before four that were almost correct, which of course is a much bigger time sink than when it's obviously wrong. Another fifteen minutes spent investigating a solution that turns out to be bogus. I got the usual gaslighting that I'm just using the wrong models, or using them wrong, etc. But when I watched others, I found that their results were just as iffy, they just usually didn't bother to check them once they looked plausible.

KaiMagnus 1y ago

I’ve been starting my prompts more and more with the phrase „Let’s brainstorm“. Really powerful seeing different options, especially based on your codebase. > I wouldn't give them a big feature again. I'll do very small things like refactoring or a very small-scoped feature. That really resonates with me. Anything larger often ends badly and I can feel the „tech debt“ building in my head with each minute Copilot is running. I do like the feeling though when you understood a problem already, write a detailed prompt to nudge the AI into the right direction, and it executes just like you wanted. After all, problem solving is why I’m here and writing code is just the vehicle for it.

i_love_retros 1y ago

Uh oh, I think the bubble is bursting. Personally the initial excitement has worn off for me and I am enjoying writing code myself and just using kagi assistant to ask the odd question, mostly research. When a team mate who bangs on about how we should all be using ai tried to demo it and got things in a bit of a mess, I knew we had peaked. And all that money invested into the hype!

href 1y ago

I've dialed down a lot as well. The answers I got for my queries were too often plain wrong. I instead started asking where I might look something up - in what man page, or in which documentation. Then I go read that. This helps me build a better mental map about where information is found (e.g., in what man page), decreasing both my reliance on search engines, and LLMs in the long run. LLMs have their uses, but they are just a tool, and an imprecise one at that.

dwoldrich 1y ago

I personally believe it's a mistake to invite AI into your editor/IDE. Keep it separate to the browser, keep discrete, concise question and answer threads. Copy and paste whenever it delivers some gold (that comes with all the copy-pasta dangers, I know - oh, don't I know it!) It's important to always maintain the developer role, don't ever surrender it.

somewhereoutth 1y ago

Unfortunately it seems that nobody 'dialed back' LLM usage for the summary on that page - a good example of how un-human such text can feel to read.

65 1y ago

LLMs require fuzzy input and are thus good for fuzzy output, mostly things like recommendations and options. I just do not see a scenario where fuzzy input can lead to absolute, opinionated output unless extremely simple and mostly done before already. Programming, design, writing, etc. all require opinions and an absolute output from the author to be quality.

Gepsens 1y ago

Llms are not a magic wand you can wave at anything and get your work cut out for you. What's new ?

piker 1y ago

Credit to the Zed team here for publishing something somewhat against its book.

nvahalik 1y ago

Sounds like he shot for the moon and missed. I've been allowing LLMs to do more "background" work for me. Giving me some room to experiment with stuff so that I can come back in 10-15 minutes and see what it's done. The key things I've come to are that it HAS to be fairly limited. Giving it a big task like refactoring a code base won't work. Giving it an example can help dramatically. If you haven't "trained" it by giving it context or adding your CLAUDE.md file, you'll end up finding it doing things you don't want it to do. Another great task I've been giving it while I'm working on other things is generating docs for existing features and modules. It is surprisingly good at looking at events and following those events to see where they go and generating diagrams and he like.

cadamsdotcom 1y ago

LLMs give you a power tool after you spent your whole career using hand tools. A chainsaw and chisel do different things and are made for different situations. It’s great to have chainsaws, no longer must we chop down a giant tree with a chisel. On the other hand there’s plenty of room in the trade for handcraft. You still need to use that chisel to smooth off the fine edges of your chainsaw work, so your teammates don’t get splinters.

jemiluv8 1y ago

When I first came across the idea of vibe coding, my first reaction was that this was taking things too far. Isn't it enough that your LLM can help you do - autocomplete - suggest possible solutions to a problem you've taken the time to understand - helps you spend less time reading documentation and instead help guide your approach and sometimes even helps you identify obscure apis that could help you get shit done - help you review your code - come up with multiple designs for a solutions - evaluate multiple designs you come up with for trade-offs - help you understand your problem better and the available apis - write a prototype of some piece of code I feel like LLMs are already doing quite a lot. I spend less time rummaging through documentation or trying to remember obscure api's or other pieces of code in a software project. All I need is a strong mental model about the project and how things are done. There is a lot of obvious heavy lifting that LLMs are doing that I for one am not able to take for granted. For people facing constraints similar to those in a resource constrained economic environment, the benefits of any technology that helps them spend less time doing work that doesn't deliver value is immediately visible/obvious/apparent. It is no longer an argument about whether it is a hype or something, it is more about how best to use it to achieve your goals. Forget the hype. Forget the marketing of AI companies - they have to do that to sell their products - nothing wrong with that. Don't let companies or bloggers set your own expectations of what could or should be done with this piece of tech. Just get on the bandwagon and experiment and find out what is too much. In the end I feel we will all come from these experiments knowing that LLMs are already doing quite a lot. TRIVIA I even came by this article https://www.greptile.com/blog/ai-code-reviews-conflict https://www.greptile.com/blog/ai-code-reviews-conflict. That clearly pointed out how LLM reliance can bring both the 10x dev and 1x dev closer to a median of "goodness". So the 10x dev is probably worse and the 1x dev ends up getting better - I'm probably that guy because I tend to mis subtle things in code and copilot review has had my ass for a while now - I haven't had defects like that in a while.

smoody07 1y ago

"Why I'm dialing back my high level language usage"

ivraatiems 1y ago

The way that LLMs are used/are encouraged by business right now is evidence that they are mostly being pushed by people who don't understand software. I don't know very many actual software engineers who advocate for vibe-coding or using LLMs this way. I know ton of engineers who advocate using them as helpful tools, myself included (in fact I changed my opinion on it as their capabilities grew, and I'll continue to do so). Every tool is just a tool. No tool is a solution. Until and unless we hit AGI, only the human brain is that.

halis 1y ago

It’s almost like the LLMs are simply a glorified autocomplete and have no actual understanding of anything they’re doing. Huh weird!

eviks 1y ago

> Alberto initially embraced LLMs with genuine enthusiasm, hoping they would revolutionize his development workflow. Is there any other way but down from such revolutionary ungrounded expectations?

matrik 1y ago

I find LLMs very efficient in a lot of things except writing big chunks of code that require some organization. They’re good at closing knowlwdge gaps, finding strategies to solve problems, writing code for defined scopes and even reviewing large PRs. However, as others said they can easily alienate you from your own project.

Amaury-El 1y ago

[dead]

careful_ai 1y ago

Too many of us fall into “prompt autopilot” mode—reaching for AI before we think. Your post calls it out beautifully: step back, reclaim the muscle memory of creative problem solving. LLMs should supplement, not substitute. That discipline often separates thoughtful integration from dependency.

deterministic 1y ago

Using AI definitely makes me more productive. However I spend 80% of my "AI time" fixing mistakes made by the AI or explaining why the solution doesn't work.

billbrown 1y ago

The use case for LLM assistance that provides value for me is solving obscure lint or static analysis warnings and errors. I take the message, provide the surrounding code, and it gives me a few approaches to solve them. More than half the time, the resolution is there and I can copy the relevant bit in the literal verbiage. (The other times it's garbage but at least I can see that this is going to require some AI—Actual Intelligence.)