15 ms·

I must be the dumbest "prompt engineer" ever, each time I ask an AI to fix or even worse, create something from scratch it rarely returns the right answer and w

I must be the dumbest "prompt engineer" ever, each time I ask an AI to fix or even worse, create something from scratch it rarely returns the right answer and when asked for modification it will struggle even more. All the incredible performance and success stories always come from these Twitter posts, I do find value in asking simple but tedious task like a small refactor or generate commands, but this "AI takes the wheel level" does not feel real.

epolanski 2y ago
I don't believe in those either, and I never see compelling YouTube videos showing that in action. For small stuff LLMs are actually great and often a lifesaver on legacy codebases, but that's more or less where it stops.
razemio 2y ago
Can you clarify what tools and programming language you use? If find that the issue often is wrong tooling, exotic programming languages or frameworks.
- deleted 2y ago
  [deleted]
- develoopest 2y ago
  I would consider frontend tasks using Typescript and React quite standard.
  - razemio 2y ago
    React in my experience sucks with AI. In fact I have not yet encountered a "heavy" framework which works well. Use something light like svelte. Typed programming languages like Typescript, Scala, Haskell and so on will produce more errors -> you need to fix stuff manually. However it will also reduce bugs. So it is a mixed bag. For an error free experience python and JavaScript work very well. When it comes to tooling, if you haven't used cline, roocode or aider (not as good) yet, you haven't seen what an AI can do. A good starter would be starting fresh, by creating a README which describes the hole application you want to build in detail and let the AI decide the tech stack. You can most certainly build complex applications with an AI in blazing speed.
branko_d 2y ago
I have the same experience. Where AI shines for me is as a form of a semantic search engine or even a tutor of sorts. I can ask for the information that I need in a relatively complex way, and more often than not it will give me a decent summary and a list of "directions" to follow-up on. If anything, it'll give me proper technical terms, that I can feed into a traditional search engine for more info. But that's never the end of my investigation and I always try to confirm the information that it gives me by consulting other sources.
- jofzar 2y ago
  Yeah I have had some "magic" moments where I knew "what" I needed, had an idea of "how it would look",but no idea how to do it and ai helped me understand how I should do it instead of the hacky very stupid way I would have done it
- mentalgear 2y ago
  Exactly same experience: since the early-access GPT-3 days, I played out various scenarios, and the most useful case has always been to use generativeAI as semantic search. It's generative features are just lacking in quality (for anything other than a toy project), and the main issues since the early GPT days remains, even though it gets better, it's still too unreliable for (mid-complex systems) serious work. Also, if you don't pay attention, it messes up other parts of the code.
- Yoric 2y ago
  Same here. In some cases, brainstorming even kinda works – I mean, it usually gives very bad responses, but it serves as a good duck. Code? Nope.
abxyz 2y ago
I think it's probably the difference between "code" and "programming". An LLM can produce code and if you're willing to surrender to the LLMs version of whatever it is you ask for, then you can have a great and productive time. If you're opinionated about programming, LLMs fall short. Most people (software engineers, developers, whatever) are not "programmers" they're "coders" which is why they have a positive impression of LLMs: they produce code, LLMs produce code... so LLMs can do a lot of their work for them. Coders used to be more productive by using libraries (e.g: don't write your own function for finding the intersection of arrays, use intersection from Lodash) whereas now libraries have been replaced by LLMs. Programmers laughed at the absurdity of left-pad[1] ("why use a dependency for 16 lines of code?") whereas coders thought left-pad was great ("why write 16 lines of code myself?"). If you think about code as a means to an end, and focus on the end, you'll get much closer to the magical experience you see spoken about on Twitter, because their acceptance criteria is "good enough" not "right". Of course, if you're a programmer who cares about the artistry of programming, that feels like a betrayal. [1] https://en.wikipedia.org/wiki/Npm_left-pad_incident https://en.wikipedia.org/wiki/Npm_left-pad_incident
- beezlewax 2y ago
  I'm waiting for artisan programming to become a thing.
  - pydry 2y ago
    Artisanal code has been a thing for a long while. If we're the luddite artisans, LLMs seem to represent the knitting frames which replaced their higher quality work with vastly cheaper, far crappier merchandise. There is a historical rhyme here.
    - ReptileMan 2y ago
      You didn't had to spend time debugging a peace of cloth, and the cloth defects are obvious
      - 2y ago
sovietmudkipz 2y ago
Maybe it’s the shot up plane effect; we only see the winners but rarely see the failures. Leads us to wrong or incorrect conclusions. Finding the right prompt to have current generation AI create the magic depicted in twitter posts may be a harder problem than most anticipate.
egorfine 2y ago
You have to learn and figure out how to prompt it. My experience with Claude Code is this: one time it produces an incredible result; another time it's an utter failure. There are prompt tips and tricks which have enormous influence on the end result.
- ido 2y ago
  Can you give us some of these tips?
  - egorfine 2y ago
    Not that I have anything concrete in my mind yet. I'm learning as we all do. But after some usage I've developed a little bit of a hunch which prompt works and which not. For example, I have mindlessly asked Claude Code over a large codebase "where is desktop app version stored and how is it presented on site". I have expected useless answer given how vague the questions was. Instead I have got a truly exceptional and extremely clear report that fully covers the question. Another example. I have asked Claude Code to come up with a script to figure out funding rate time intervals on a given exchange and Code ended up in an almost endless loop running small test scripts in node.js to figure this out and came up with a super suboptimal and complicated solution. Turns out my prompt was too verbose and detailed and I have specifically asked Claude Code to figure out time intervals, not just get them. So it did. Instead of just querying the exchange via API and printing the list on terminal (3 lines script) it actually, truly tried to figure them out in various ways.
  - borgdefenser 2y ago
    You should also try the same prompt multiple times to see how this works. Sometimes you will get better or worse answers completely by chance. I think Claude is pretty good if you have it write a function and give it the inputs, output and a data example. You can also put to ask clarifying questions as needed because there is a good chance there are aspects of the prompt that are ambiguous. My prompts are always better if I write them in a separate text file and then paste them in too. I think I just take my time and think things out more that way instead of trying to get to the answer as fast.
moomin 2y ago
I can definitely save time, but I find I need to be very precise about the exact behaviour, a skill I learned as… a regular programmer. Soper up is higher in languages I’m not familiar with, where I know what needs doing but not necessarily the standard way to do it.
fullstackwife 2y ago
"wild", "insane" keywords usually are a good filter for marketing spam.
- belter 2y ago
  Influencer would be another term...
smallerfish 2y ago
I've done code interviews with hundreds of candidates recently. The difference between those who are using LLMs effectively and those who are not is stark. I honestly think engineers who think like OP are going to get left behind. Take a weekend to work on getting your head around this by building a personal project (or learning a new language). A few things to note: a) Use the "Projects" feature in Claude web. The context makes a significant amount of difference in the output. Curate what it has in the context; prune out old versions of files and replace them. This is annoying UX, yes, but it'll give you results. b) Use the project prompt to customize the response. E.g. I usually tell it not to give me redundant code that I already have. (Claude can otherwise be overly helpful and go on long riffs spitting out related code, quickly burning through your usage credits). c) If the initial result doesn't work, give it feedback and tell it what's broken (build messages, descriptions of behavior, etc). d) It's not perfect. Don't give up if you don't get perfection.
- jacobedawson 2y ago
  I'd add to that that the best results are with clear spec sheets, which you can create using Claude (web) or another model like ChatGPT or Grok. Telling them what you want and what tech you're using helps them create a technical description with clear segments and objectives, and in my experience works wonders in getting Claude Code on the right track, where it has full access to the entire context of your code base.
- triyambakam 2y ago
  Hundreds of candidates? That's significant if not an exaggeration. What are the stark differences you have seen? Did you inquire about the candidate's use of language models?
  - nsonha 2y ago
    if it's real that person interviewed at least one candidate per day last year. Idk what kind of engineering role in what kind of org where you even do that.
    - 2y ago
yodsanklai 2y ago
> I do find value in asking simple but tedious task like a small refactor or generate commands, This is already a productivity boost. I'm more and more impressed about what I can get out of these tools (as you said, simple but tedious things). ChatGPT4o (provided by company) does pretty complex things for me, and I use it more and more. Actually, I noticed that when I can't use it (e.g. internal tools/languages), I'm pretty frustrated.
- cglace 2y ago
  Are you concerned that these tools will soon replace the need for engineers?
  - yodsanklai 2y ago
    Yes, I used to be skeptical about the hype, but now I'm somewhat concerned. I don't think they will replace engineers but they do increase their productivity. I'm not able to quantify by how much though. In my case, maybe it increases my productivity by 5-10%, saving me a few hours of work each week. Very rough estimate. Does it mean that we'll need less engineers to perform the same amount of work? or we'll produce better products? In my company, there's no shortage of things to do, so I don't think we'll hire less people if suddenly engineers are a bit more productive. But who knows how it'll impact the industry as a whole.
julienmarie 2y ago
I initially had the same experience. My codebase is super opinionated with a specific way to handle things. Initially it kept on wanting to do things it's way. I then changed my approach and documented the way the codebase is structured, how things should be done, all the conventions used and on every prompt I make sure to tell him to use these documents as reference. I also have a central document that keeps track of dependencies of modules and the global data model. Since I made these documents as reference developing new features has been a breathe. I created the architecture, documented it, and now it uses it. The way I prompt it is first I write the documentation of the module I want, following the format I detailed inbthe master documents, and ask him to follow the documentation and specs. I use cursor as well, but more as an assistant when I work on the architecture pieces. But I would never let an AI the driver seat for building the architecture and making tech decisions.
csomar 2y ago
Yeah, it's so bad now I only trust my eyes. Everyone is faking posts, tweets and benchmarks that the truth no longer exists. I'm using Claude 3.7 now and while it improved on certain areas, it degraded on others (ie: it randomly removes/changes things more now).
- namaria 2y ago
  It's clear to anyone paying attention that LLMs hit a wall a while back. RAG is just expert systems with extra steps. 'Reasoning' is just burning more tokens in hopes it somehow makes the results better. And lately we've seen that a small blanket is being pulled that way or another. LLMs are cool, machine learning is cooler. Still no 'AI' in sight.
slooonz 2y ago
I decided to try seriously the Sonnet 3.7. I started with a simple prompt on claude.ai ("Do you know claude code ? Can you do a simple implementation for me ?"). After minimal tweaking from me, it gave me this : https://gist.github.com/sloonz/3eb7d7582c33e95f2b000a092001614c#file-version1-py https://gist.github.com/sloonz/3eb7d7582c33e95f2b000a0920016... After interacting with this tool, I decided it would be nice if the tool could edit itself, so I asked (him ? it ?) to create its next version. It came up with a non-working version of this https://gist.github.com/sloonz/3eb7d7582c33e95f2b000a092001614c#file-version2-py https://gist.github.com/sloonz/3eb7d7582c33e95f2b000a0920016.... I fixed the bug manually, but it started an interactive loop : I could now describe what I wanted, describe the bugs, and the tool will add the features/fix the bugs itself. I decided to rewrite it in Typescript (by that I mean: can you rewrite yourself in typescript). And then add other tools (by that: create tools and unit tests for the tools). https://gist.github.com/sloonz/3eb7d7582c33e95f2b000a092001614c#file-edit-file-tool-ts https://gist.github.com/sloonz/3eb7d7582c33e95f2b000a0920016... and https://gist.github.com/sloonz/3eb7d7582c33e95f2b000a092001614c#file-edit-file-tests-ts https://gist.github.com/sloonz/3eb7d7582c33e95f2b000a0920016... have been created by the tool itself, without any manual fix from me. Setting up the testing/mock framework ? Done by the tool itself too. In one day (and $20), I essentially had recreated claude-code. That I could improve just by asking "Please add feature XXX". $2 a feature, with unit tests, on average.
- Silhouette 2y ago
  Thanks for writing up your experience and sharing the real code. It is fascinating to see how close these tools can now get to producing useful, working software by themselves. That said - I'm wary of reading too much into results at this scale. There isn't enough code in such a simple application to need anything more sophisticated than churning out a few lines of boilerplate that produce the correct result. It probably won't be practical for the current state of the art in code generators to write large-scale production applications for a while anyway just because of the amount of CPU time and RAM they'd need. But assuming we solve the performance issues one way or another eventually it will be interesting to see whether the same kind of code generators can cope with managing projects at larger scales where usually the hard problems have little to do with efficiently churning out boilerplate code.
crabl 2y ago
What I've noticed from my extensive use over the past couple weeks has been Claude Code really sucks at thinking things through enough to understand the second and third order consequences of the code that it's writing. That said, it's easy enough to work around its deficiencies by using a model with extended thinking (Grok, GPT4.5, Sonnet 3.7 in thinking mode) to write prompts for it and use Claude Code as basically a dumb code-spewing minion. My workflow has been: give Grok enough context on the problem with specific code examples, ask it to develop an implementation plan that a junior developer can follow, and paste the result into Claude Code, asking it to diligently follow the implementation plan and nothing else.
- cglace 2y ago
  In all of these posts I fail to see how this is engineering anymore. It seems like we are one step away from taking ourselves out of the picture completely.
  - bckr 2y ago
    I don’t write binaries, assembly, or C. If I don’t have to write an application, I’m okay with that. I still have to write the requirements, design, and acceptance criteria. I still have to gather the requirements from stakeholders, figure out why those will or will not work, provision infra, figure out how to glue said infra together, test and observe and debug the whole thing, get feedback from stakeholders… I have plenty of other stuff to do. And if you automate 99% of the above work? Then the requirements are going to get 100Xed. Put all the bells and whistles in. Make it break the laws of physics. Make it never ever crash and always give incredibly detailed feedback to the end users. Make it beautiful and faster than thought itself. I’m not worried about taking myself out of the loop.
    - cglace 2y ago
      Thanks for sharing. I hope you are right. It's hard to stay objective as things are changing so quickly.
    - draebek 2y ago
matt_heimer 2y ago
LLM are replacing Google for me when coding. When I want to get something implemented, let's say make a REST request in Java using a specific client library, I previously used Google to find example of using that library. Google has gotten worse (or the internet has more garbage) so finding code an example is more difficult than it used to be. Now I ask an LLM for an example. Sometimes I have to ask for a refinement and and usually something is broken in the example but it takes less time to get the LLM produced example to work than it does to find a functional example using Google. But the LLM has only replaced my previous Google usage, I didn't expect Google to develop my applications and I don't with LLMs.
- ptmcc 2y ago
  This has been my experience of successful usage as well. It's not writing code for me, but pulling together the equivalent of a Stack Overflow example and some explaining sentences that I can follow up on. Not perfect and I don't blindly copy paste it, same as Stack Overflow ever was, but faster and more interactive. It's helpful for wayfinding, but not producing the end result.
- layer8 2y ago
  In order to use a library, I need to (this is my opinion) be able to reason about the library’s behavior, based on a specification of its interface contract. The LLM may help with coming up with suitable code, but verifying that the application logic is correct with respect to the library’s documented interface contract is still necessary. It’s therefore still a requirement to read and understand the library’s documentation. For example, for the case of a REST client, you need to understand how the possible failure modes of the HTTP protocol and REST API are translated by the library.
- deergomoo 2y ago
  I used the Kagi free trial when I was doing Advent of Code in a somewhat unfamiliar language (Swift) last year, as well as ChatGPT occasionally. The LLM was obviously much faster and the information was much higher density, but it had quite literally about a 20% rate of just making up APIs from my limited experiment. But I was very impressed with Kagi’s results and ended up signing up, now using it as my primary search engine.
Kiro 2y ago
Funny, because I have the same feeling toward the "I never get it to work" comments. You don't need any special prompt engineering so that's definitely not it.
nsonha 2y ago
this space is moving really fast, I suggest before forming an definitive opinion try the best tool, such as the latest Claude model and use "agentic" mode or the equivalence on your client. For example, on Copilot this mode is brand new and only available in vscode insider. Cursor and other tools have had it for a little longer.
- collingreen 2y ago
  People have been saying it writes amazing code that works for far longer than that setup has been available though. Your comment makes me think the product is still trying to catch up to these expectations people are setting. That being said I appreciate your suggestion and will consider giving that a shot.
Delomomonl 2y ago
I had Claude prototype a few things and for that it's really enjoyable. Like a single page HTML J's page which does a few things and saves it state in local storage with a json backup feature (download the json). I also enjoy it for doing things I don't care much but makes it more polished. Like I hate my basically empty readme with two commands. It looks ugly and when I come back to stuff like this a few days/weeks later I always hate it. Claude just generates really good readmes. I'm trying out Claude code right now and like it so far.
Balgair 2y ago
Hey, I've been hearing about this issue that programmers have on HN a lot. But I'm in the more 'bad programmer/hacker' camp and think that LLMs are amazing and really helpful. I know that one can post a link to the chat history. Can you do that for an example that you are comfortable sharing? I know that it may not be possible though or very time consuming. What I'm trying to get at is: I suck at programming, I know that. And you probably suck a lot less. And if you say that LLMs are garbage, and I say they are great, I want to know where I'm getting the disconnect. I'm sincerely, not trying to be a troll here, and I really do want to learn more. Others are welcome to post examples and walk through them too. Thanks for any help here.
- vlod 2y ago
  >and think that LLMs are amazing and really helpful Respectively, are you understanding what it produces or do you think that's its amazing because it produces something, that 'maybe' works. Here's an e.g. I was just futzing with. I did a refactor of my code (typescript) and my test code broke (vitest) and for some reason it said 'mockResolvedValue()' is not a function. I've done this a gazillion times. I allowed it via 3-4 iterations to try and fix it (I was being lazy and wanted my error to go away) and the amount of crap (rewriting tests, referenced code) it was producing was beyond ridiculous. (I was using github co-pilot). Eventually I said "f.that for a game of soldiers" and used by brain. I forgot to uncomment a vi.mock() during the refactor. I DO use it to fix stupid typescript errors (the error blob it dumps on you can be a real pita to process) and appreciate it when gives me a simple solution. So I agree with quite a few comments here. I'm not ready to bend the knee to our AI Overloads.
  - Balgair 2y ago
    Yeah, so I'm a 'hacker' (MIT definition here). I've only taken a single class in javascript with Sun based workstations about 20 years ago now (god, I'm old). I hated it. All my work now is in python and SQL now and though I've watched a lot of youtube videos and plunked at StackOverFlow for ~15 years, I've never had formal education in either language. Like, it takes me about as long to set up the libraries and dependancies in python as it does to write my code. My formal job titles have never had 'programmer' in them. My code, as such, is just to get something done. Mostly this is hardware interfacing stuff, but a little software too. I've had my code get passed up the chain and incorporated into codebases, but that's only happened a handful of times. The vast majority of the code I write is never seen by others. My code hasn't had to be maintainable as I've never really had to maintain it for more than 5 years max. I've used git on projects before, but I don't really see the need these days. The largest program I've written is ~1M lines of code. Most code is about 100 lines long now. I almost always know what 'working' means, in that I know the output that I want to see (again, mostly working in hardware). I almost never run tests of the code. I've had the same issues you have had with LLMs, where they get stuck and I have to try to restart the process. Usually this happens to me in about 20 back and forths. I'm mostly just pasting relevant code snippets and the errors back into the LLM for a while until things work for me. Again, I know what 'working' looks like from the start. Typically, I'll start the session with an LLM by telling it the problem I have, what I want the code skeleton to look like, and then what I want the output to look like. Then it will give the psuedo code, then I walk it through each portion of the psuedo code. Then I get to errors and debugging. Usually about half of this is just libraries and versions in python. Then, I get to the errors of the code itself. I can typically find what line of code is causing the error just from the terminal outputs. I'll talk with the LLM about that line of code, trying to understand it from the error. Then, on to the next error. Repeat that process until I get the working output I desire. I'm never expecting the right code out of the first LLM interaction, but I am expecting (and seeing) that the time it takes to get to working code is faster. The time it would usually take me to get through all this before LLMs was about 2 weeks of work (~80 hours) per project. Now it takes me about half a day (~4 hours), and it's getting faster. I'm not in the camp of thinking that AI is going to take my job. I am in the camp of thinking that AI is going to finally let me burn down the list of things that we really need to do around here. Thank you for the reply!
kolbe 2y ago
I am willing to say I am a good prompt engineer, and "AI takes the wheel" is only ever my experience when my task is a very easy one. AI is fantastic for a few elements of the coding process--building unit tests, error checking, deciphering compile errors, and autocompleting trivially repetitive sections. But I have not been able to get it "take the wheel"
iambateman 2y ago
I have a challenging, repetitive developer task that I need to do ~200 times. It’s for scraping a site and getting similar pieces of data. I wrote a worksheet for Cursor and give it specific notes for how to accomplish the task in a particular case. Then let it run and it’s fairly successful. Keep in mind…it’s never truly “hands off” for me. I still need to clean things up after it’s done. But it’s very good at figuring out how to filter the HTML down and parse out the data I need. Plus it writes good tests. So my success story is that it takes 75% of the energy out of a task I find particularly tedious.
- WD-42 2y ago
  I haven’t found llm code gen to be very good except in cases like you mention here. When you need to do large boilerplatey code with a lot of hardcoded values or parameters. The kind of thing you could probably write a code generator yourself for if you cared enough to do it. Thankfully Llms can save us from some of that.
clusterhacks 2y ago
I am a slow adopter of new tech but plan to spend a significant amount of time in 2025 using AI tools when coding. I am net negative on AI simply replacing programmers, but I think the practice of development is undergoing a seismic shift at this point. My recent usage is oriented towards using pseudocode descriptions that closely map to Python to produce Python functions. I am very impressed with Claude 3.7's syntactic correctness when given a chunk of pseudocode that looks "python-y" to begin with. My one concern is that much of my recent code requirements lack novelty. So there is a somewhat reasonable chance that the tool is just spitting out code it slurped somewhere in github or elsewhere in the larger Internet. Just this week, I gave Claude a relatively "anonymous" function in pseudocode, meaning variable names were not particularly descriptive with one tiny exception. However, Claude generated a situationally appropriate comment as part of the function definition. This was . . . surprising to me if somehow the model had NOT in its training set had some very close match to my pseudocode description that included enough context to add the comment.
- doug_durham 2y ago
  At this point very little code is "novel". Everyone is simply rewriting code that has already been written in a similar form. The LLM isn't slurping up and restating code verbatim. It is taking code that it has seen thousands of times and generating a customized version for your needs. It's hubris to think that anyone here is generating "novel" code.
  - clusterhacks 2y ago
    I have seen the argument that very little code is novel but I find it inherently unsatisfying and lacking in nuance? I think what bugs me about is that if you squint hard enough, all programming reduces to "take some data, do something to it." That "something" is doing a lot of heavy lifting in the argument that "something" is or isn't novel. Heck, if we think about it from the programming language perspective, all code is "simply" using already existing language functions to cobble together a solution to some specific set of requirements. Is no program novel? There is probably a consideration here that maybe boils down to the idea of engineering vs artisanal craftsmanship and where a specific project falls in that spectrum . . .
noufalibrahim 2y ago
I'm in the same boat. I've found it useful in micro contexts but in larger programs, it's like a "yes man" that just agrees with what I suggest and creates an implementation without considering the larger ramifications. I don't know if it's just me.
BeetleB 2y ago
Some hints for people stuck like this: Consider using Aider. It's a great tool and cheaper to use than Code. Look at Aiders LLM leaderboard to figure out which LLMs to use. Use its architect mode (although you can get quite fast without it - I personally haven't needed it). Work incrementally. I use at least 3 branches. My main one, a dev one and a debug one. I develop on dev. When I encounter a bug I switch to debug. The reason is it can produce a lot of code to fix a bug. It will write some code to fix it. That won't work. It will try again and write even more code. Repeat until fixed. But in the end I only needed a small subset of the new code. So you then revert all the changes and have it fix it again telling it the correct fix. Don't debug on your dev branch. Aider's auto committing is scary but really handy. Limit your context to 25k. Only add files that you think are necessary. Combining the two: Don't have large files. Add a Readme.md file. It will then update the file as it makes code changes. This can give you a glimpse of what it's trying to do and if it writes something problematic you know it's not properly understanding your goal. Accept that it is not you and will write code differently from you. Think of it as a moderately experienced coder who is modifying the codebase. It's not going to follow all your conventions. https://aider.chat/ https://aider.chat/ https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/
- tptacek 2y ago
  The three-branch thing is so smart.
  - BeetleB 2y ago
    It took a while for me to realize it, and frankly, it's kind of embarrassing that I didn't think of it immediately. It is, after all, what many of us would do in our manual SW development. But when using an LLM that seems pretty good, we just assume we don't need to follow all the usual good practices.
    - vlovich123 2y ago
      Does the LLM make commits along the way? I think I’m missing why you need all these branches vs git reset —hard once it figures out the bug?
escapecharacter 2y ago
I've found AI to be useful on precisely-scoped tasks I might assign to a junior programmer to take a day to do, like "convert this exact bash script to a Powershell script". But in my own work, those tasks are pretty rare, like 3 times a month? Often I start working on something, and the scope and definition of success changes while I'm in the midst of it. Or it turns out to be harder than expected and it makes sense to timebox it and do a quick search for workarounds. As much as we joke about StackOverflow commenters sometimes telling a question-asker they shouldn't be doing what they're trying to do, you do actually want that (soft) pushback some of the time. Most modern LLMs will gleefully come up with a complete plan for how we're gonna get the square peg in the round hole.
- deleted 2y ago
  [deleted]
- philipswood 2y ago
  > you do actually want that (soft) pushback some of the time. Most modern LLMs will gleefully come up with a complete plan for how we're gonna get the square peg in the round hole. I once accidentally asked a local DeepSeek distilled model to do the wrong thing by accidentally copy pasting the wrong variable name. It told me how to do it, and then asked me if I was sure. My local DeepSeek R1 model (deepseek-r1:7b) saw me trying to do something stuupid (I was working with the wrong variable). It told me how to do what I asked and then asked: > _Is this modification part of a larger change you're making to the code? I'd like to make sure we're not modifying something important that might have side effects._ Looking at its though process: > _The user wants to modify this string by replacing "-input" with "-kb". But looking at the ARN structure, I recall that AWS S3 ARNs are usually in the form arn<:aws:1151472526310103070>s3:::bucket_name RegionalPart path. The part after the bucket name is typically s3:// followed by the object key._ > _Wait, maybe the user has a specific reason to make this change. Perhaps they're formatting or structuring something for a different purpose. They might not realize that directly modifying ARNs can cause issues elsewhere if not done correctly._
  - escapecharacter 2y ago
Ancalagon 2y ago
Are you actually using claude? There's an enormous difference between claude code and copilot, with the latter being a bigger burden these days than a help.
EigenLord 2y ago
You've got to do piecemeal validation steps yourself, especially for models like Sonnet 3.7 that tend to over-generate code and bury themselves in complexity. Windsurf seems to be onto something. Running Sonnet 3.7 in thinking mode will sometimes reveal bits and pieces about the prompts they're injecting when it mentions "ephemeral messages" reminding it about what files it recently visited. That's all external scaffolding and context built around the model to keep it on track.
babyent 2y ago
I’ve dug into this a few times. Every single time they were doing something simple. Just because someone has decades of experience or is a SME in some niche doesn’t mean they’re actually good… engineers.
timewizard 2y ago
Oh. That's because he's clearly lying.
dilap 2y ago
Yeah I gave Claude Code a try at about 5 different things, with miserable results on all of them (insult to injury -- each time it charged me about a buck!). I wonder if because it was C# with Unity code, maybe not so heavily represented in the training set? I still find lots of use for LLMs authoring stuff at more like the function level. "I know I need exactly this." Edit: I did however find it amazing for asking questions about sections of the code I did not write.
InvertedRhodium 2y ago
My workflow for that kind of thing goes something like this (I use Sonnet 3.7 Thinking in Cursor): 1. 1st prompt is me describing what I want to build, what I know I want and any requirements or restrictions I'm aware of. Based on these requirements, ask a series of questions to produce a complete specification document. 2. Workshop the specification back and forward until I feel it's complete enough. 3. Ask the agent to implement the specification we came up with. 4. Tell the agent to implement Cursor Rules based on the specifications to ensure consistent implementation details in future LLM sessions. I'd say it's pretty good 80% of the time. You definitely still need to understand the problem domain and be able to validate the work that's been produced but assuming you had some architectural guidelines you should be able to follow the code easily. The Cursor Rules step makes all the difference in my experience. I picked most of this workflow up from here: https://ghuntley.com/stdlib/ https://ghuntley.com/stdlib/ Edit: A very helpful rule is to tell Cursor to always checkout a new branch based on the latest HEAD of master/main for all of it's work.
- theshrike79 2y ago
  I need to steal the specification idea. Cursor w/ Claude has a habit of running away on tangents instead of solving just the one problem, then I need to reject its changes and even roll back to a previous version. With a proper specification as guideline it might stay on track a bit better.
  - prettyblocks 2y ago
    Copilot supports this somewhat natively: https://docs.github.com/en/copilot/customizing-copilot/adding-repository-custom-instructions-for-github-copilot https://docs.github.com/en/copilot/customizing-copilot/addin... The first thing I do for a new project is ask Copilot to create a custom-instructions.md for me and then as I work on my projects, I ask it to update the instructions every now and then based on the current state of my project. Much less misses this way in my experience.
    - theshrike79 2y ago
_steve_yegge_ 2y ago
Gene and I would like to invite you to review our book, if you're up for it. It should be ready for early review in about 7-10 days. It seems like you would be the perfect audience for it. We're hoping the book can teach you what you need in order to have all those success stories yourself.
- Gunnerhead 2y ago
  How can I follow this book? I’m interested too.
gloosx 2y ago
From the creators of static open-source marketing benchmarks: twitter PR posts.
rhubarbtree 2y ago
I spent a few hours trying cursor. I was impressed at first, I liked the feel of it and I tried to vibe code, whatever that means. I tried to get it to build a very simple version of an app I’ve been working on. But the basics didn’t work, and as I got it to fix some functionality other stuff broke. It repeatedly nuked the entire web app, then rolled back again and again. It tried quick and dirty solutions that would lead to dead ends in just a few more features. No sense of elegance or foundational abstractions. The code it produced was actually OK, and I could have fixed the bugs given enough time, but overall the results were far inferior to every programmer I’ve ever worked with. On the design side, the app was ugly as hell and I couldn’t get it to fix that at all. Autocomplete on a local level seems far more useful.
dgellow 2y ago
I have the same experience
onion2k 2y ago
it rarely returns the right answer One of the biggest difficulties AI will face is getting developers to unlearn the idea that there's a right answer, and that of the many thousands of possible right answers, 'the code I would have written myself' is just one (or a few if you're one of the few great devs who don't stop thinking about approaches after your first attempt.)
habinero 2y ago
It's not. A lot -- and I mean a lot -- of people who hype it up are hobby or aspirational coders. If you drill down on what exactly they use it for, they invariably don't write code in professional settings that will be maintained and which other humans have to read. Everyone who does goes "eh, it's good for throwaway code or one offs and it's decent at code completion". Then there's the "AGI will doom us all" cult weirdos, but we don't talk about them.
darepublic 2y ago
What model are you using
jayd16 2y ago
I agree it feels very different from my experience. I'm curious when we'll start seeing verifiable results like live code streams with impressive results or companies dominating the competition with AI built products.
huvin99 2y ago
Is this true even for Claude 3.7 Sonnet/3.7 Sonnet Thinking ?
deleted 2y ago
[deleted]

epolanski 2y ago

I don't believe in those either, and I never see compelling YouTube videos showing that in action. For small stuff LLMs are actually great and often a lifesaver on legacy codebases, but that's more or less where it stops.

razemio 2y ago

Can you clarify what tools and programming language you use? If find that the issue often is wrong tooling, exotic programming languages or frameworks.

deleted 2y ago
[deleted]
develoopest 2y ago
I would consider frontend tasks using Typescript and React quite standard.
- razemio 2y ago
  React in my experience sucks with AI. In fact I have not yet encountered a "heavy" framework which works well. Use something light like svelte. Typed programming languages like Typescript, Scala, Haskell and so on will produce more errors -> you need to fix stuff manually. However it will also reduce bugs. So it is a mixed bag. For an error free experience python and JavaScript work very well. When it comes to tooling, if you haven't used cline, roocode or aider (not as good) yet, you haven't seen what an AI can do. A good starter would be starting fresh, by creating a README which describes the hole application you want to build in detail and let the AI decide the tech stack. You can most certainly build complex applications with an AI in blazing speed.

branko_d 2y ago

I have the same experience. Where AI shines for me is as a form of a semantic search engine or even a tutor of sorts. I can ask for the information that I need in a relatively complex way, and more often than not it will give me a decent summary and a list of "directions" to follow-up on. If anything, it'll give me proper technical terms, that I can feed into a traditional search engine for more info. But that's never the end of my investigation and I always try to confirm the information that it gives me by consulting other sources.

jofzar 2y ago
Yeah I have had some "magic" moments where I knew "what" I needed, had an idea of "how it would look",but no idea how to do it and ai helped me understand how I should do it instead of the hacky very stupid way I would have done it
mentalgear 2y ago
Exactly same experience: since the early-access GPT-3 days, I played out various scenarios, and the most useful case has always been to use generativeAI as semantic search. It's generative features are just lacking in quality (for anything other than a toy project), and the main issues since the early GPT days remains, even though it gets better, it's still too unreliable for (mid-complex systems) serious work. Also, if you don't pay attention, it messes up other parts of the code.
Yoric 2y ago
Same here. In some cases, brainstorming even kinda works – I mean, it usually gives very bad responses, but it serves as a good duck. Code? Nope.

abxyz 2y ago

I think it's probably the difference between "code" and "programming". An LLM can produce code and if you're willing to surrender to the LLMs version of whatever it is you ask for, then you can have a great and productive time. If you're opinionated about programming, LLMs fall short. Most people (software engineers, developers, whatever) are not "programmers" they're "coders" which is why they have a positive impression of LLMs: they produce code, LLMs produce code... so LLMs can do a lot of their work for them. Coders used to be more productive by using libraries (e.g: don't write your own function for finding the intersection of arrays, use intersection from Lodash) whereas now libraries have been replaced by LLMs. Programmers laughed at the absurdity of left-pad[1] ("why use a dependency for 16 lines of code?") whereas coders thought left-pad was great ("why write 16 lines of code myself?"). If you think about code as a means to an end, and focus on the end, you'll get much closer to the magical experience you see spoken about on Twitter, because their acceptance criteria is "good enough" not "right". Of course, if you're a programmer who cares about the artistry of programming, that feels like a betrayal. [1] https://en.wikipedia.org/wiki/Npm_left-pad_incident https://en.wikipedia.org/wiki/Npm_left-pad_incident

beezlewax 2y ago
I'm waiting for artisan programming to become a thing.
- pydry 2y ago
  Artisanal code has been a thing for a long while. If we're the luddite artisans, LLMs seem to represent the knitting frames which replaced their higher quality work with vastly cheaper, far crappier merchandise. There is a historical rhyme here.
  - ReptileMan 2y ago
    You didn't had to spend time debugging a peace of cloth, and the cloth defects are obvious
    - 2y ago

sovietmudkipz 2y ago

Maybe it’s the shot up plane effect; we only see the winners but rarely see the failures. Leads us to wrong or incorrect conclusions. Finding the right prompt to have current generation AI create the magic depicted in twitter posts may be a harder problem than most anticipate.

egorfine 2y ago

You have to learn and figure out how to prompt it. My experience with Claude Code is this: one time it produces an incredible result; another time it's an utter failure. There are prompt tips and tricks which have enormous influence on the end result.

ido 2y ago
Can you give us some of these tips?
- egorfine 2y ago
  Not that I have anything concrete in my mind yet. I'm learning as we all do. But after some usage I've developed a little bit of a hunch which prompt works and which not. For example, I have mindlessly asked Claude Code over a large codebase "where is desktop app version stored and how is it presented on site". I have expected useless answer given how vague the questions was. Instead I have got a truly exceptional and extremely clear report that fully covers the question. Another example. I have asked Claude Code to come up with a script to figure out funding rate time intervals on a given exchange and Code ended up in an almost endless loop running small test scripts in node.js to figure this out and came up with a super suboptimal and complicated solution. Turns out my prompt was too verbose and detailed and I have specifically asked Claude Code to figure out time intervals, not just get them. So it did. Instead of just querying the exchange via API and printing the list on terminal (3 lines script) it actually, truly tried to figure them out in various ways.
- borgdefenser 2y ago
  You should also try the same prompt multiple times to see how this works. Sometimes you will get better or worse answers completely by chance. I think Claude is pretty good if you have it write a function and give it the inputs, output and a data example. You can also put to ask clarifying questions as needed because there is a good chance there are aspects of the prompt that are ambiguous. My prompts are always better if I write them in a separate text file and then paste them in too. I think I just take my time and think things out more that way instead of trying to get to the answer as fast.

moomin 2y ago

I can definitely save time, but I find I need to be very precise about the exact behaviour, a skill I learned as… a regular programmer. Soper up is higher in languages I’m not familiar with, where I know what needs doing but not necessarily the standard way to do it.

fullstackwife 2y ago

"wild", "insane" keywords usually are a good filter for marketing spam.

belter 2y ago
Influencer would be another term...

smallerfish 2y ago

I've done code interviews with hundreds of candidates recently. The difference between those who are using LLMs effectively and those who are not is stark. I honestly think engineers who think like OP are going to get left behind. Take a weekend to work on getting your head around this by building a personal project (or learning a new language). A few things to note: a) Use the "Projects" feature in Claude web. The context makes a significant amount of difference in the output. Curate what it has in the context; prune out old versions of files and replace them. This is annoying UX, yes, but it'll give you results. b) Use the project prompt to customize the response. E.g. I usually tell it not to give me redundant code that I already have. (Claude can otherwise be overly helpful and go on long riffs spitting out related code, quickly burning through your usage credits). c) If the initial result doesn't work, give it feedback and tell it what's broken (build messages, descriptions of behavior, etc). d) It's not perfect. Don't give up if you don't get perfection.

jacobedawson 2y ago
I'd add to that that the best results are with clear spec sheets, which you can create using Claude (web) or another model like ChatGPT or Grok. Telling them what you want and what tech you're using helps them create a technical description with clear segments and objectives, and in my experience works wonders in getting Claude Code on the right track, where it has full access to the entire context of your code base.
triyambakam 2y ago
Hundreds of candidates? That's significant if not an exaggeration. What are the stark differences you have seen? Did you inquire about the candidate's use of language models?
- nsonha 2y ago
  if it's real that person interviewed at least one candidate per day last year. Idk what kind of engineering role in what kind of org where you even do that.
  - 2y ago

yodsanklai 2y ago

> I do find value in asking simple but tedious task like a small refactor or generate commands, This is already a productivity boost. I'm more and more impressed about what I can get out of these tools (as you said, simple but tedious things). ChatGPT4o (provided by company) does pretty complex things for me, and I use it more and more. Actually, I noticed that when I can't use it (e.g. internal tools/languages), I'm pretty frustrated.

cglace 2y ago
Are you concerned that these tools will soon replace the need for engineers?
- yodsanklai 2y ago
  Yes, I used to be skeptical about the hype, but now I'm somewhat concerned. I don't think they will replace engineers but they do increase their productivity. I'm not able to quantify by how much though. In my case, maybe it increases my productivity by 5-10%, saving me a few hours of work each week. Very rough estimate. Does it mean that we'll need less engineers to perform the same amount of work? or we'll produce better products? In my company, there's no shortage of things to do, so I don't think we'll hire less people if suddenly engineers are a bit more productive. But who knows how it'll impact the industry as a whole.

julienmarie 2y ago

I initially had the same experience. My codebase is super opinionated with a specific way to handle things. Initially it kept on wanting to do things it's way. I then changed my approach and documented the way the codebase is structured, how things should be done, all the conventions used and on every prompt I make sure to tell him to use these documents as reference. I also have a central document that keeps track of dependencies of modules and the global data model. Since I made these documents as reference developing new features has been a breathe. I created the architecture, documented it, and now it uses it. The way I prompt it is first I write the documentation of the module I want, following the format I detailed inbthe master documents, and ask him to follow the documentation and specs. I use cursor as well, but more as an assistant when I work on the architecture pieces. But I would never let an AI the driver seat for building the architecture and making tech decisions.

csomar 2y ago

Yeah, it's so bad now I only trust my eyes. Everyone is faking posts, tweets and benchmarks that the truth no longer exists. I'm using Claude 3.7 now and while it improved on certain areas, it degraded on others (ie: it randomly removes/changes things more now).

namaria 2y ago
It's clear to anyone paying attention that LLMs hit a wall a while back. RAG is just expert systems with extra steps. 'Reasoning' is just burning more tokens in hopes it somehow makes the results better. And lately we've seen that a small blanket is being pulled that way or another. LLMs are cool, machine learning is cooler. Still no 'AI' in sight.

slooonz 2y ago

I decided to try seriously the Sonnet 3.7. I started with a simple prompt on claude.ai ("Do you know claude code ? Can you do a simple implementation for me ?"). After minimal tweaking from me, it gave me this : https://gist.github.com/sloonz/3eb7d7582c33e95f2b000a092001614c#file-version1-py https://gist.github.com/sloonz/3eb7d7582c33e95f2b000a0920016... After interacting with this tool, I decided it would be nice if the tool could edit itself, so I asked (him ? it ?) to create its next version. It came up with a non-working version of this https://gist.github.com/sloonz/3eb7d7582c33e95f2b000a092001614c#file-version2-py https://gist.github.com/sloonz/3eb7d7582c33e95f2b000a0920016.... I fixed the bug manually, but it started an interactive loop : I could now describe what I wanted, describe the bugs, and the tool will add the features/fix the bugs itself. I decided to rewrite it in Typescript (by that I mean: can you rewrite yourself in typescript). And then add other tools (by that: create tools and unit tests for the tools). https://gist.github.com/sloonz/3eb7d7582c33e95f2b000a092001614c#file-edit-file-tool-ts https://gist.github.com/sloonz/3eb7d7582c33e95f2b000a0920016... and https://gist.github.com/sloonz/3eb7d7582c33e95f2b000a092001614c#file-edit-file-tests-ts https://gist.github.com/sloonz/3eb7d7582c33e95f2b000a0920016... have been created by the tool itself, without any manual fix from me. Setting up the testing/mock framework ? Done by the tool itself too. In one day (and $20), I essentially had recreated claude-code. That I could improve just by asking "Please add feature XXX". $2 a feature, with unit tests, on average.

Silhouette 2y ago
Thanks for writing up your experience and sharing the real code. It is fascinating to see how close these tools can now get to producing useful, working software by themselves. That said - I'm wary of reading too much into results at this scale. There isn't enough code in such a simple application to need anything more sophisticated than churning out a few lines of boilerplate that produce the correct result. It probably won't be practical for the current state of the art in code generators to write large-scale production applications for a while anyway just because of the amount of CPU time and RAM they'd need. But assuming we solve the performance issues one way or another eventually it will be interesting to see whether the same kind of code generators can cope with managing projects at larger scales where usually the hard problems have little to do with efficiently churning out boilerplate code.

crabl 2y ago

What I've noticed from my extensive use over the past couple weeks has been Claude Code really sucks at thinking things through enough to understand the second and third order consequences of the code that it's writing. That said, it's easy enough to work around its deficiencies by using a model with extended thinking (Grok, GPT4.5, Sonnet 3.7 in thinking mode) to write prompts for it and use Claude Code as basically a dumb code-spewing minion. My workflow has been: give Grok enough context on the problem with specific code examples, ask it to develop an implementation plan that a junior developer can follow, and paste the result into Claude Code, asking it to diligently follow the implementation plan and nothing else.

cglace 2y ago
In all of these posts I fail to see how this is engineering anymore. It seems like we are one step away from taking ourselves out of the picture completely.
- bckr 2y ago
  I don’t write binaries, assembly, or C. If I don’t have to write an application, I’m okay with that. I still have to write the requirements, design, and acceptance criteria. I still have to gather the requirements from stakeholders, figure out why those will or will not work, provision infra, figure out how to glue said infra together, test and observe and debug the whole thing, get feedback from stakeholders… I have plenty of other stuff to do. And if you automate 99% of the above work? Then the requirements are going to get 100Xed. Put all the bells and whistles in. Make it break the laws of physics. Make it never ever crash and always give incredibly detailed feedback to the end users. Make it beautiful and faster than thought itself. I’m not worried about taking myself out of the loop.
  - cglace 2y ago
    Thanks for sharing. I hope you are right. It's hard to stay objective as things are changing so quickly.
  - draebek 2y ago

matt_heimer 2y ago

LLM are replacing Google for me when coding. When I want to get something implemented, let's say make a REST request in Java using a specific client library, I previously used Google to find example of using that library. Google has gotten worse (or the internet has more garbage) so finding code an example is more difficult than it used to be. Now I ask an LLM for an example. Sometimes I have to ask for a refinement and and usually something is broken in the example but it takes less time to get the LLM produced example to work than it does to find a functional example using Google. But the LLM has only replaced my previous Google usage, I didn't expect Google to develop my applications and I don't with LLMs.

ptmcc 2y ago
This has been my experience of successful usage as well. It's not writing code for me, but pulling together the equivalent of a Stack Overflow example and some explaining sentences that I can follow up on. Not perfect and I don't blindly copy paste it, same as Stack Overflow ever was, but faster and more interactive. It's helpful for wayfinding, but not producing the end result.
layer8 2y ago
In order to use a library, I need to (this is my opinion) be able to reason about the library’s behavior, based on a specification of its interface contract. The LLM may help with coming up with suitable code, but verifying that the application logic is correct with respect to the library’s documented interface contract is still necessary. It’s therefore still a requirement to read and understand the library’s documentation. For example, for the case of a REST client, you need to understand how the possible failure modes of the HTTP protocol and REST API are translated by the library.
deergomoo 2y ago
I used the Kagi free trial when I was doing Advent of Code in a somewhat unfamiliar language (Swift) last year, as well as ChatGPT occasionally. The LLM was obviously much faster and the information was much higher density, but it had quite literally about a 20% rate of just making up APIs from my limited experiment. But I was very impressed with Kagi’s results and ended up signing up, now using it as my primary search engine.

Kiro 2y ago

Funny, because I have the same feeling toward the "I never get it to work" comments. You don't need any special prompt engineering so that's definitely not it.

nsonha 2y ago

this space is moving really fast, I suggest before forming an definitive opinion try the best tool, such as the latest Claude model and use "agentic" mode or the equivalence on your client. For example, on Copilot this mode is brand new and only available in vscode insider. Cursor and other tools have had it for a little longer.

collingreen 2y ago
People have been saying it writes amazing code that works for far longer than that setup has been available though. Your comment makes me think the product is still trying to catch up to these expectations people are setting. That being said I appreciate your suggestion and will consider giving that a shot.

Delomomonl 2y ago

I had Claude prototype a few things and for that it's really enjoyable. Like a single page HTML J's page which does a few things and saves it state in local storage with a json backup feature (download the json). I also enjoy it for doing things I don't care much but makes it more polished. Like I hate my basically empty readme with two commands. It looks ugly and when I come back to stuff like this a few days/weeks later I always hate it. Claude just generates really good readmes. I'm trying out Claude code right now and like it so far.

Balgair 2y ago

Hey, I've been hearing about this issue that programmers have on HN a lot. But I'm in the more 'bad programmer/hacker' camp and think that LLMs are amazing and really helpful. I know that one can post a link to the chat history. Can you do that for an example that you are comfortable sharing? I know that it may not be possible though or very time consuming. What I'm trying to get at is: I suck at programming, I know that. And you probably suck a lot less. And if you say that LLMs are garbage, and I say they are great, I want to know where I'm getting the disconnect. I'm sincerely, not trying to be a troll here, and I really do want to learn more. Others are welcome to post examples and walk through them too. Thanks for any help here.

vlod 2y ago
>and think that LLMs are amazing and really helpful Respectively, are you understanding what it produces or do you think that's its amazing because it produces something, that 'maybe' works. Here's an e.g. I was just futzing with. I did a refactor of my code (typescript) and my test code broke (vitest) and for some reason it said 'mockResolvedValue()' is not a function. I've done this a gazillion times. I allowed it via 3-4 iterations to try and fix it (I was being lazy and wanted my error to go away) and the amount of crap (rewriting tests, referenced code) it was producing was beyond ridiculous. (I was using github co-pilot). Eventually I said "f.that for a game of soldiers" and used by brain. I forgot to uncomment a vi.mock() during the refactor. I DO use it to fix stupid typescript errors (the error blob it dumps on you can be a real pita to process) and appreciate it when gives me a simple solution. So I agree with quite a few comments here. I'm not ready to bend the knee to our AI Overloads.
- Balgair 2y ago
  Yeah, so I'm a 'hacker' (MIT definition here). I've only taken a single class in javascript with Sun based workstations about 20 years ago now (god, I'm old). I hated it. All my work now is in python and SQL now and though I've watched a lot of youtube videos and plunked at StackOverFlow for ~15 years, I've never had formal education in either language. Like, it takes me about as long to set up the libraries and dependancies in python as it does to write my code. My formal job titles have never had 'programmer' in them. My code, as such, is just to get something done. Mostly this is hardware interfacing stuff, but a little software too. I've had my code get passed up the chain and incorporated into codebases, but that's only happened a handful of times. The vast majority of the code I write is never seen by others. My code hasn't had to be maintainable as I've never really had to maintain it for more than 5 years max. I've used git on projects before, but I don't really see the need these days. The largest program I've written is ~1M lines of code. Most code is about 100 lines long now. I almost always know what 'working' means, in that I know the output that I want to see (again, mostly working in hardware). I almost never run tests of the code. I've had the same issues you have had with LLMs, where they get stuck and I have to try to restart the process. Usually this happens to me in about 20 back and forths. I'm mostly just pasting relevant code snippets and the errors back into the LLM for a while until things work for me. Again, I know what 'working' looks like from the start. Typically, I'll start the session with an LLM by telling it the problem I have, what I want the code skeleton to look like, and then what I want the output to look like. Then it will give the psuedo code, then I walk it through each portion of the psuedo code. Then I get to errors and debugging. Usually about half of this is just libraries and versions in python. Then, I get to the errors of the code itself. I can typically find what line of code is causing the error just from the terminal outputs. I'll talk with the LLM about that line of code, trying to understand it from the error. Then, on to the next error. Repeat that process until I get the working output I desire. I'm never expecting the right code out of the first LLM interaction, but I am expecting (and seeing) that the time it takes to get to working code is faster. The time it would usually take me to get through all this before LLMs was about 2 weeks of work (~80 hours) per project. Now it takes me about half a day (~4 hours), and it's getting faster. I'm not in the camp of thinking that AI is going to take my job. I am in the camp of thinking that AI is going to finally let me burn down the list of things that we really need to do around here. Thank you for the reply!

kolbe 2y ago

I am willing to say I am a good prompt engineer, and "AI takes the wheel" is only ever my experience when my task is a very easy one. AI is fantastic for a few elements of the coding process--building unit tests, error checking, deciphering compile errors, and autocompleting trivially repetitive sections. But I have not been able to get it "take the wheel"

iambateman 2y ago

I have a challenging, repetitive developer task that I need to do ~200 times. It’s for scraping a site and getting similar pieces of data. I wrote a worksheet for Cursor and give it specific notes for how to accomplish the task in a particular case. Then let it run and it’s fairly successful. Keep in mind…it’s never truly “hands off” for me. I still need to clean things up after it’s done. But it’s very good at figuring out how to filter the HTML down and parse out the data I need. Plus it writes good tests. So my success story is that it takes 75% of the energy out of a task I find particularly tedious.

WD-42 2y ago
I haven’t found llm code gen to be very good except in cases like you mention here. When you need to do large boilerplatey code with a lot of hardcoded values or parameters. The kind of thing you could probably write a code generator yourself for if you cared enough to do it. Thankfully Llms can save us from some of that.

clusterhacks 2y ago

I am a slow adopter of new tech but plan to spend a significant amount of time in 2025 using AI tools when coding. I am net negative on AI simply replacing programmers, but I think the practice of development is undergoing a seismic shift at this point. My recent usage is oriented towards using pseudocode descriptions that closely map to Python to produce Python functions. I am very impressed with Claude 3.7's syntactic correctness when given a chunk of pseudocode that looks "python-y" to begin with. My one concern is that much of my recent code requirements lack novelty. So there is a somewhat reasonable chance that the tool is just spitting out code it slurped somewhere in github or elsewhere in the larger Internet. Just this week, I gave Claude a relatively "anonymous" function in pseudocode, meaning variable names were not particularly descriptive with one tiny exception. However, Claude generated a situationally appropriate comment as part of the function definition. This was . . . surprising to me if somehow the model had NOT in its training set had some very close match to my pseudocode description that included enough context to add the comment.

doug_durham 2y ago
At this point very little code is "novel". Everyone is simply rewriting code that has already been written in a similar form. The LLM isn't slurping up and restating code verbatim. It is taking code that it has seen thousands of times and generating a customized version for your needs. It's hubris to think that anyone here is generating "novel" code.
- clusterhacks 2y ago
  I have seen the argument that very little code is novel but I find it inherently unsatisfying and lacking in nuance? I think what bugs me about is that if you squint hard enough, all programming reduces to "take some data, do something to it." That "something" is doing a lot of heavy lifting in the argument that "something" is or isn't novel. Heck, if we think about it from the programming language perspective, all code is "simply" using already existing language functions to cobble together a solution to some specific set of requirements. Is no program novel? There is probably a consideration here that maybe boils down to the idea of engineering vs artisanal craftsmanship and where a specific project falls in that spectrum . . .

noufalibrahim 2y ago

I'm in the same boat. I've found it useful in micro contexts but in larger programs, it's like a "yes man" that just agrees with what I suggest and creates an implementation without considering the larger ramifications. I don't know if it's just me.

BeetleB 2y ago

Some hints for people stuck like this: Consider using Aider. It's a great tool and cheaper to use than Code. Look at Aiders LLM leaderboard to figure out which LLMs to use. Use its architect mode (although you can get quite fast without it - I personally haven't needed it). Work incrementally. I use at least 3 branches. My main one, a dev one and a debug one. I develop on dev. When I encounter a bug I switch to debug. The reason is it can produce a lot of code to fix a bug. It will write some code to fix it. That won't work. It will try again and write even more code. Repeat until fixed. But in the end I only needed a small subset of the new code. So you then revert all the changes and have it fix it again telling it the correct fix. Don't debug on your dev branch. Aider's auto committing is scary but really handy. Limit your context to 25k. Only add files that you think are necessary. Combining the two: Don't have large files. Add a Readme.md file. It will then update the file as it makes code changes. This can give you a glimpse of what it's trying to do and if it writes something problematic you know it's not properly understanding your goal. Accept that it is not you and will write code differently from you. Think of it as a moderately experienced coder who is modifying the codebase. It's not going to follow all your conventions. https://aider.chat/ https://aider.chat/ https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/

tptacek 2y ago
The three-branch thing is so smart.
- BeetleB 2y ago
  It took a while for me to realize it, and frankly, it's kind of embarrassing that I didn't think of it immediately. It is, after all, what many of us would do in our manual SW development. But when using an LLM that seems pretty good, we just assume we don't need to follow all the usual good practices.
  - vlovich123 2y ago
    Does the LLM make commits along the way? I think I’m missing why you need all these branches vs git reset —hard once it figures out the bug?

escapecharacter 2y ago

I've found AI to be useful on precisely-scoped tasks I might assign to a junior programmer to take a day to do, like "convert this exact bash script to a Powershell script". But in my own work, those tasks are pretty rare, like 3 times a month? Often I start working on something, and the scope and definition of success changes while I'm in the midst of it. Or it turns out to be harder than expected and it makes sense to timebox it and do a quick search for workarounds. As much as we joke about StackOverflow commenters sometimes telling a question-asker they shouldn't be doing what they're trying to do, you do actually want that (soft) pushback some of the time. Most modern LLMs will gleefully come up with a complete plan for how we're gonna get the square peg in the round hole.

deleted 2y ago
[deleted]
philipswood 2y ago
> you do actually want that (soft) pushback some of the time. Most modern LLMs will gleefully come up with a complete plan for how we're gonna get the square peg in the round hole. I once accidentally asked a local DeepSeek distilled model to do the wrong thing by accidentally copy pasting the wrong variable name. It told me how to do it, and then asked me if I was sure. My local DeepSeek R1 model (deepseek-r1:7b) saw me trying to do something stuupid (I was working with the wrong variable). It told me how to do what I asked and then asked: > _Is this modification part of a larger change you're making to the code? I'd like to make sure we're not modifying something important that might have side effects._ Looking at its though process: > _The user wants to modify this string by replacing "-input" with "-kb". But looking at the ARN structure, I recall that AWS S3 ARNs are usually in the form arn<:aws:1151472526310103070>s3:::bucket_name RegionalPart path. The part after the bucket name is typically s3:// followed by the object key._ > _Wait, maybe the user has a specific reason to make this change. Perhaps they're formatting or structuring something for a different purpose. They might not realize that directly modifying ARNs can cause issues elsewhere if not done correctly._
- escapecharacter 2y ago

Ancalagon 2y ago

Are you actually using claude? There's an enormous difference between claude code and copilot, with the latter being a bigger burden these days than a help.

EigenLord 2y ago

You've got to do piecemeal validation steps yourself, especially for models like Sonnet 3.7 that tend to over-generate code and bury themselves in complexity. Windsurf seems to be onto something. Running Sonnet 3.7 in thinking mode will sometimes reveal bits and pieces about the prompts they're injecting when it mentions "ephemeral messages" reminding it about what files it recently visited. That's all external scaffolding and context built around the model to keep it on track.

babyent 2y ago

I’ve dug into this a few times. Every single time they were doing something simple. Just because someone has decades of experience or is a SME in some niche doesn’t mean they’re actually good… engineers.

timewizard 2y ago

Oh. That's because he's clearly lying.

dilap 2y ago

Yeah I gave Claude Code a try at about 5 different things, with miserable results on all of them (insult to injury -- each time it charged me about a buck!). I wonder if because it was C# with Unity code, maybe not so heavily represented in the training set? I still find lots of use for LLMs authoring stuff at more like the function level. "I know I need exactly this." Edit: I did however find it amazing for asking questions about sections of the code I did not write.

InvertedRhodium 2y ago

My workflow for that kind of thing goes something like this (I use Sonnet 3.7 Thinking in Cursor): 1. 1st prompt is me describing what I want to build, what I know I want and any requirements or restrictions I'm aware of. Based on these requirements, ask a series of questions to produce a complete specification document. 2. Workshop the specification back and forward until I feel it's complete enough. 3. Ask the agent to implement the specification we came up with. 4. Tell the agent to implement Cursor Rules based on the specifications to ensure consistent implementation details in future LLM sessions. I'd say it's pretty good 80% of the time. You definitely still need to understand the problem domain and be able to validate the work that's been produced but assuming you had some architectural guidelines you should be able to follow the code easily. The Cursor Rules step makes all the difference in my experience. I picked most of this workflow up from here: https://ghuntley.com/stdlib/ https://ghuntley.com/stdlib/ Edit: A very helpful rule is to tell Cursor to always checkout a new branch based on the latest HEAD of master/main for all of it's work.

theshrike79 2y ago
I need to steal the specification idea. Cursor w/ Claude has a habit of running away on tangents instead of solving just the one problem, then I need to reject its changes and even roll back to a previous version. With a proper specification as guideline it might stay on track a bit better.
- prettyblocks 2y ago
  Copilot supports this somewhat natively: https://docs.github.com/en/copilot/customizing-copilot/adding-repository-custom-instructions-for-github-copilot https://docs.github.com/en/copilot/customizing-copilot/addin... The first thing I do for a new project is ask Copilot to create a custom-instructions.md for me and then as I work on my projects, I ask it to update the instructions every now and then based on the current state of my project. Much less misses this way in my experience.
  - theshrike79 2y ago

_steve_yegge_ 2y ago

Gene and I would like to invite you to review our book, if you're up for it. It should be ready for early review in about 7-10 days. It seems like you would be the perfect audience for it. We're hoping the book can teach you what you need in order to have all those success stories yourself.

Gunnerhead 2y ago
How can I follow this book? I’m interested too.

gloosx 2y ago

From the creators of static open-source marketing benchmarks: twitter PR posts.

rhubarbtree 2y ago

I spent a few hours trying cursor. I was impressed at first, I liked the feel of it and I tried to vibe code, whatever that means. I tried to get it to build a very simple version of an app I’ve been working on. But the basics didn’t work, and as I got it to fix some functionality other stuff broke. It repeatedly nuked the entire web app, then rolled back again and again. It tried quick and dirty solutions that would lead to dead ends in just a few more features. No sense of elegance or foundational abstractions. The code it produced was actually OK, and I could have fixed the bugs given enough time, but overall the results were far inferior to every programmer I’ve ever worked with. On the design side, the app was ugly as hell and I couldn’t get it to fix that at all. Autocomplete on a local level seems far more useful.

dgellow 2y ago

I have the same experience

onion2k 2y ago

it rarely returns the right answer One of the biggest difficulties AI will face is getting developers to unlearn the idea that there's a right answer, and that of the many thousands of possible right answers, 'the code I would have written myself' is just one (or a few if you're one of the few great devs who don't stop thinking about approaches after your first attempt.)

habinero 2y ago

It's not. A lot -- and I mean a lot -- of people who hype it up are hobby or aspirational coders. If you drill down on what exactly they use it for, they invariably don't write code in professional settings that will be maintained and which other humans have to read. Everyone who does goes "eh, it's good for throwaway code or one offs and it's decent at code completion". Then there's the "AGI will doom us all" cult weirdos, but we don't talk about them.

darepublic 2y ago

What model are you using

jayd16 2y ago

I agree it feels very different from my experience. I'm curious when we'll start seeing verifiable results like live code streams with impressive results or companies dominating the competition with AI built products.

huvin99 2y ago

Is this true even for Claude 3.7 Sonnet/3.7 Sonnet Thinking ?

deleted 2y ago

[deleted]