9 ms·

LLMs are mirrors of operator skill

47 points by ghuntley 1y ago

ninetyninenine 1y ago
My hope is that the AI never improves to take over my job and the next generation of programmers is so used to learning programming with AI that they become mirrors of hallucinating AI. That eliminates ageism and AI taking my job. Realistically though I think AI will come to a point where it can take over my job. But if not, this is my hope.
- whatnow37373 1y ago
  Maybe we can learn some lessons from digital artists who naturally fret over the use of their skills and how they will be replaced by stable diffusion and friends. In one way, yes, this massively shifts power into the hands of the less skilled. On the other hand if you’d need some proper and I mean proper marketing materials, who are you going to hire? A professional artist using AI or some dipshit with AI? There will be slop of course but after a while everyone has slop and the only differentiating factor will be quality or at least some gate-kept arbitrary level of complexity. Like how rich people want fancy hand made stuff. Edit: my point is mainly that the level will rise to a point that you’d need to be scientist to create a - then - fancy app again. You see this with web. It was easy, we made it ridiculous and I mean ridiculously complicated where you need to study computer science to debug React rendering for your marketing pamphlet.
mouse_ 1y ago
isn't the whole selling point of ML the idea that operators no longer need to be as skilled? it feels like the goal posts have been moving as of late.
- breckenedge 1y ago
  Sort of. The super users will massively outcompete unskilled users. So while it does raise everyone’s abilities, it will be lopsided
  - disgruntledphd2 1y ago
    I mean, all the research (to be fair there's very little) suggests that in CS (customer support) the major gains are for people who are worse than others, and there was little to no impact for the higher skilled cs people. And that's kinda what one would expect, given that LLMs are basically a blurry JPEG of the web/github etc. Like, I think reasoning can help here, but I rarely see good results when prompting an LLM with something complicated (technical statistical problems and good approaches) while they are fantastic at less edge case stuff (working with docker and well known frameworks). So yeah, definite productivity gains but I'm not convinced that they're as transformational as they are being pitched.
  - gmerc 1y ago
    That's clearly why the VCs are now pushing 996 and everyone has to work twice as hard. Lol
- IggleSniggle 1y ago
  That's the selling point to VCs. The selling point to consumers of the tech is that AI will make only them, with their special inherent qualities, outperform their peers even more (unspoken, but also implying that if they are an underperformer they might be able to catch up), but only if they act now! This limited time opportunity will soon be used by everyone!
- whartung 1y ago
  Isn't the whole point of <abstraction> that we don't need to worry about the inner details? A friend currently has an AI workflow that pits two AIs against each other. They start with an issues database. AI 1 pulls issues, and fixes them, and commits them. AI 2 reviews the work and makes new issues. And the cycle repeats. After a bunch of work is complete and issues are flagged as done, the tests run green, he grabs all of the commits, walks through them, cleans them up if necessary, and crunches them into one big commit to the main branch. He loves waking up in the morning to a screen filled with completed things. He has, essentially, self-promoted to management over some potentially untrustworthy junior developers with occasional flashes of brilliance. And I was pondering that, and it just reminded me of something I've been feeling for sometime. A lot of modern development is wiring together large, specialized libraries. Our job is to take these things (along with their cascade of dependencies) and put our own little bits of glue and logic around them. And, heck, if I wanted to glue badly documented black boxes together, I would have gone into Electrical Engineering. But here's the thing. While the layers are getting thicker, the abstractions more opaque, in the end, much like a CEO is responsible for that ONE PERSON down in the factory behaving badly, we, as users of this stuff, are responsible for ALL OF IT. Down to bugs in the microprocessor. When push comes to shove, it's all on us. We can whine and complain and delegate. "I didn't write the compiler, not my fault." "Not my library..." "The Manager assured the VP who assured me that..." But doesn't really matter when you have a smoking crater of a system, does it? Because we're the ones delivering services and such, putting our names on it. So, yea, no, you don't have to be "as skilled", perhaps, when using something. But you're still responsible for it.
  - ghuntley 1y ago
    Absolutely! I kind of hate the term "vibe coding" because of its associations with brain off. It is so important for an engineer to take accountability for what they ship. Now to your ponderoo about libraries, something I've found that's really fascinating is I've really stopped using open source libraries unless there's a network ecosystem effect for example like Tailwind. But for everything else, it's much easier to code generate it. And if there's something wrong with the implementation, I can take ownership and accountability for that implementation and I can just fix it with a couple more prompts. No more bullshit in open source. Some person might not even be maintaining the project anymore or having to deal with like getting a pull request fixed or open source supply chain attack vectors related to project takeovers and all that noise. It just doesn't exist anymore. It's really changed how I do software development.
jbellis 1y ago
Yes! This is an underappreciated point that both sides in "AI makes coding better" vs "AI just writes slop" have mostly missed. Co-authoring code with AI is a qualitatively different activity than writing code by hand, and it requires new skills. I started Brokk to give humans better tools with which to do this: https://github.com/BrokkAi/brokk https://github.com/BrokkAi/brokk
- ghuntley 1y ago
  Hey dude, it's been a couple weeks since we caught up for a zoom. Maybe it's three weeks now. Still keen to catch up again, dude. It's gonna be a little bit busy for the next couple of weeks. I've got two conference talks to do then I'm gonna be over in San Fran, but keen.
elliotbnvl 1y ago
> If they waste time by not using the debugger, adding debug log statements, or writing tests, then they're not a good fit. Do you mean writing manual tests? Because having the LLM write tests is key in iteration speed w/o backtracking.
- ghuntley 1y ago
  Typo on my part. I completely agree. Back pressure is everything. Edit: I've just updated the post.
  - elliotbnvl 1y ago
    Nice. FWIW, since I just realized my only main comment was a criticism, I found your article very insightful. It baffles me how many people will disagree with the general premise or nit-pick one tiny detail. The only thing more surprising to me than the rate at which AI is developing is the number of developers jamming their heads into the sand over it.
    - ghuntley 1y ago
      Ah, I wrote a blog post specifically about this. Some engineers are not going to make it. https://ghuntley.com/ngmi https://ghuntley.com/ngmi
      - elliotbnvl 1y ago
        Oh, I read this the other day on HN and agreed strongly. I hadn't realized this was the same blog. Nice!
- foldr 1y ago
  The scope of the ‘not’ is also unclear here. I think they’re saying that you waste time if you don’t do all the things in the list, but it could also be read as saying that not using the debugger is a waste of time, and that adding debug log statements and writing tests is also a waste of time.
- 1y ago
bgwalter 1y ago
"Someone can be highly experienced as a software engineer in 2024, but that does not mean they're skilled as a software engineer in 2025, now that AI is here." With that arrogance, my only question is: Where is your own code and what makes you more qualified than Linux kernel or gcc developers?
- elliotbnvl 1y ago
  It's not arrogance, since they're not asserting anything about themself. It's a factual observation with which you're free to disagree – but should be challenged directly rather than via ad hominem if you want to actually make a point rather than just collect internet snark kudos. Also, there are far more generic web developers than there are Linux kernel developers, and they represent the vast majority of the market share / profit generation in software development, so your metric isn't really relevant either.
  - skydhash 1y ago
    So what has changed about the realm of programming that make all the skill obsolete, including the skill to learn new programming thingies? The DOM API is old, All the mainstream backend languages are old, unix administrations has barely changed (only the way to use those tools have). Even Elasticsearch is 15 years old. Amazon S3 is past drinking age in many countries around the world. And that's just pertaining to web projects. You just need to open a university textbook to realize how old many of the fundamentals are. Most shiny new things is old stuff repackaged.
    - elliotbnvl 1y ago
      A lot of people are rejecting AI because of how transformational it is. Those people will fall behind the people who adopt it aggressively. It's akin to people who refused to learn C because they knew assembly.
      - skydhash 1y ago
        I don't think people has refused to learn C (which is not particularly hard to learn for someone who knows about assembly and various other languages at the time). A lot of compilers were buggy and people have lots of assembly snippets for particular routines. And that's not counting mature projects that was already in assembly and you have to maintain. A lot of programmers are actually fine trying new stuff out, but some are professionals and don't bring everything under the sun in their work projects.
brcmthrowaway 1y ago
Is there a good guide resource on properly using LLMs/agents for coding? How to get started?
- ghuntley 1y ago
  See the tweet on workflow that I put in the post. No courseware, no bullshit, it's there. Have fun. The blog has plenty of guidance from using specs to creating standard libraries of prompts and how to clone a venture capital-backed company while you sleep.
- spacebanana7 1y ago
  I found that learning prompt engineering was largely a waste of time. The value of the knowledge seems to depreciate so quickly. I spent loads of time learning to use special syntax that helped GPT 3.5 or comfy ui for Stable diffusion. Now the latest models can do exactly what I want without any of those “high skill” prompts. The context windows are so big that we can be quite lazy about dumping files into prompts without optimisation. The only general advice I’d give it to take more risk and continually ask more of the models.
mromanuk 1y ago
Every time I ask an LLM to write some UI and model for SwiftUI I have to specify to use @Observable macro (is the new way), which they normally do, after asking for it. The LLM tells me that they prefere the "older way" because it's more broadly compatible, that's ok if you are aiming for that. But If the programmer doesn't know about that they will be stuck with the LLM calling the shots for them.
- maxwell 1y ago
  It's just a higher level abstraction, subject to leaks as with all abstractions. How many professional programmers don't have assemblers/compilers/interpreters "calling the shots" on arbitrary implementation details outside the problem domain?
  - LorenPechtel 1y ago
    But we trust those tools to do the job correctly. The compiler has considerable latitude in messing with the details so long as the result is guaranteed to match what was ordered--when we find any deviation from that even in an edge case we consider it a bug (Borland Pascal debugger, I'm looking at you. I wasted a *lot* of time on the fact that in single step mode you peacefully "execute" an invalid segment register load!) LLMs lack this guarantee.
    - maxwell 1y ago
      We trust those tools to do the job correctly now. https://vivekhaldar.com/articles/when-compilers-were-the--ai--that-scared-programmers https://vivekhaldar.com/articles/when-compilers-were-the--ai...
- cpinto 1y ago
  Have you tried writing rules for how you want things done, instead of repeating the same things every time?
- starlust2 1y ago
  A thing people miss is that there are many different right ways to solve a problem. A legacy system might need the compatibility or it might be a greenfield. If you leave a technical requirement out of the prompt you are letting the LLM decide. Maybe that will agree with your nuanced view of things, but maybe not. We're not yet at a point where LLM coders will learn all your idiosyncrasies automatically, but those feedback loops are well within our technical ability. LLM's are roughly a knowledgeable but naïve junior dev; you must train them! Hint: add that requirement to your system/app prompt and be done with it.
namuol 1y ago
> If I were interviewing a candidate now, the first things I'd ask them to explain would be the fundamentals of how the Model Context Protocol works and how to build an agent. Um, what? This is the sort of knowledge you definitely do not need in your back pocket. It’s literally the perfect kind of question for AI to answer. Also this is such a moving target that I suspect most hiring processes change at a slower pace.
- dgfitz 1y ago
  I would just give some sort of LLM-esque answer that sounds correct but is very wrong, and hope I would get the opportunity to follow that up with: "Oh, I must have hallucinated that, can you give me a better prompt?" I crack myself up.
  - layer8 1y ago
    Don’t forget to apologize and acknowledge that they are right.
    - a_e_k 1y ago
      You're right to ask about that! ...
- elliotbnvl 1y ago
  It's a good heuristic for determining how read in somebody is to the current AI space, which is super important right now, regardless of being a moving target. The actual understanding of MCP is less important than the mindset having such an understanding represents.
  - namuol 1y ago
    Hard disagree. It’s not super important to be AI-pilled. You just need to be a good communicator. The tooling is a moving target, but so long as you can explain what you need well and can identify confusion or hallucination, you’ll be effective with them.
foldr 1y ago
Isn’t this just a roundabout way of saying that people who are skilled at using LLMs will get better results from them than people who aren’t? In other words, LLMs “mirror operator skill” in about the same way as hammers, paintbrushes or any other tool. Hardly a blinding insight.
- energy123 1y ago
  It's a controversial opinion because both AI optimists and AI pessimists can find room for disagreement with the premise. The optimists think vibe coding is about to be fully automated and humans don't have long, one or two years at best. The pessimists think LLMs don't add much value in the first place. In either case they would disagree with the premise.
  - marinmania 1y ago
    Agreed. By HN standards I am a very shitty programmer, and as of a year ago I would have said it takes up about 25% of my time. I pretty much just make demos to display some non-coding research. I think with the rise of LLMs, my coding time has been cut down by almost half. And I definitely need to bring in help less often. In that sense it has raised my floor, while making the people above me (not necessarily super coders, but still more advanced) less needed.
makmanalp 1y ago
Counterthoughts: a) These skills fit on a double sided sheet of paper (e.g. the claude code best practices doc) and b) what these skills are has been changing so rapidly that even the best practices docs fall out of date super quick. For example, managing the context window has become less of a problem with increased context windows in newer models and tools like the auto-resummarization / context window refresh in claude code make it so that you might be just fine without doing anything yourself. All this to say that the idea that you're left significantly behind if you aren't training yourself on this feels bogus (I say this as a person who /does/ use these tools daily). It should take any programmer not more than a few hours to learn these skills from scratch, with the help of a doc, meaning any employee you hire should be able to pick these up no problem. I'm not sure it makes sense as a hiring filter. Perhaps in the future this will change. But right now these tools are built more like user friendly appliances - more like a cellphone or a toaster than a technology to wrap your head around, like a compiler or a database.
patrickhogan1 1y ago
I agree with most of what you’re saying—especially the Unix pipe analogy and the value of building a prompt library while understanding which LLM to use. That said, I think there’s value in a catch-all fallback: running a prompt without all the usual rules or assumptions. Sometimes, a simple prompt on the latest model just works—and often in surprisingly more effective ways than one with a complex prompt.
henning 1y ago
Centering an interview around MCP and "building an agent" which often means "write a prompt and make HTTP calls to some LLM host service" is incredibly stupid unless that is what the product the company produces is. MCP is a tool and may only be minor in terms of relevance to a position. If I use tools that use MCP but our actual business is about something else, the interview should be about what the company actually does. Your arrogance and presumptions about the future don't make you look smart when you are so likely to be wrong. Enumerating enough predictions until one of them is right is not prescient, it's bullshit.
jmsdnns 1y ago
A key thing that has been shown in research at Wharton is that LLMs elevate people with less experience a lot more than they elevate people a lots of experience. If we take "operator skill" to mean "they know how to make prompts", there is some truth to it and we can see that by whether or not the operator is designing the context window or not. But for the more important question, whether or not LLMs are useful has an inverse relationship with how skilled the person is in the context they are using for. This is why the best engineers mostly shrug at LLMs while those that arent the best feel a big lift. So, LLMs are not mirrors of operator skill. This post is instead an argument that everyone should become prompt engineers.
- ghuntley 1y ago
  Keen to read the research. Can you drop the link?
  - jmsdnns 1y ago
    I was at Penn's first AI conference last year and heard Dr Lilach Mollick's keynote where she said this is shown to be true over and over. She doesnt seem to publish often, but her husband Ethan always has a lot to say about AI. https://www.oneusefulthing.org/p/everyone-is-above-average https://www.oneusefulthing.org/p/everyone-is-above-average
    - ghuntley 1y ago
      Thanks.
- namuol 1y ago
  Disagree. Poor engineers will go in circles with AI because they will under specify their tasks and fail to recognize improper solutions. Ultimately if you’re not thoughtful about your problem and critical about the solution you will fail. This is true with or without AI at the wheel.
  - Jensson 1y ago
deleted 1y ago
[deleted]
deleted 1y ago
[deleted]
satisfice 1y ago
A lot of these questions no one knows the answer to. If you think you know “the best LLM to use for summarizing” then you must have done a whole lot of expensive testing— but you didn’t, did you? At best you saw a comment on HN and you believed it. And if you did do such testing, I hope it wasn’t more than a month ago, because it’s out of date, now. The nature of my job affords me the luxury of playing with AI tech to do a lot of things, including helping me write code. But I’m not able to answer detailed technical questions about which LLM is best for what. There is no reliable and durable data. The situation itself is changing too quickly to track unless you have the resources to have full subscriptions to everything and you don’t have any real work to do.
incomingpain 1y ago
I have a greater appreciation of LLMs after working out how to do them offline. My 2 year old graphics card sure could be bigger... kicks can... it was plenty for starfield... holy crap vram is expensive! >If they waste time by not using the debugger, not adding debug log statements, or failing to write tests, then they're not a good fit. Why would you deprive yourself of a tool that will make you better?

47 points by ghuntley 1y ago

My hope is that the AI never improves to take over my job and the next generation of programmers is so used to learning programming with AI that they become mirrors of hallucinating AI. That eliminates ageism and AI taking my job. Realistically though I think AI will come to a point where it can take over my job. But if not, this is my hope.

whatnow37373 1y ago
Maybe we can learn some lessons from digital artists who naturally fret over the use of their skills and how they will be replaced by stable diffusion and friends. In one way, yes, this massively shifts power into the hands of the less skilled. On the other hand if you’d need some proper and I mean proper marketing materials, who are you going to hire? A professional artist using AI or some dipshit with AI? There will be slop of course but after a while everyone has slop and the only differentiating factor will be quality or at least some gate-kept arbitrary level of complexity. Like how rich people want fancy hand made stuff. Edit: my point is mainly that the level will rise to a point that you’d need to be scientist to create a - then - fancy app again. You see this with web. It was easy, we made it ridiculous and I mean ridiculously complicated where you need to study computer science to debug React rendering for your marketing pamphlet.

mouse_ 1y ago

isn't the whole selling point of ML the idea that operators no longer need to be as skilled? it feels like the goal posts have been moving as of late.

breckenedge 1y ago
Sort of. The super users will massively outcompete unskilled users. So while it does raise everyone’s abilities, it will be lopsided
- disgruntledphd2 1y ago
  I mean, all the research (to be fair there's very little) suggests that in CS (customer support) the major gains are for people who are worse than others, and there was little to no impact for the higher skilled cs people. And that's kinda what one would expect, given that LLMs are basically a blurry JPEG of the web/github etc. Like, I think reasoning can help here, but I rarely see good results when prompting an LLM with something complicated (technical statistical problems and good approaches) while they are fantastic at less edge case stuff (working with docker and well known frameworks). So yeah, definite productivity gains but I'm not convinced that they're as transformational as they are being pitched.
- gmerc 1y ago
  That's clearly why the VCs are now pushing 996 and everyone has to work twice as hard. Lol
IggleSniggle 1y ago
That's the selling point to VCs. The selling point to consumers of the tech is that AI will make only them, with their special inherent qualities, outperform their peers even more (unspoken, but also implying that if they are an underperformer they might be able to catch up), but only if they act now! This limited time opportunity will soon be used by everyone!
whartung 1y ago
Isn't the whole point of <abstraction> that we don't need to worry about the inner details? A friend currently has an AI workflow that pits two AIs against each other. They start with an issues database. AI 1 pulls issues, and fixes them, and commits them. AI 2 reviews the work and makes new issues. And the cycle repeats. After a bunch of work is complete and issues are flagged as done, the tests run green, he grabs all of the commits, walks through them, cleans them up if necessary, and crunches them into one big commit to the main branch. He loves waking up in the morning to a screen filled with completed things. He has, essentially, self-promoted to management over some potentially untrustworthy junior developers with occasional flashes of brilliance. And I was pondering that, and it just reminded me of something I've been feeling for sometime. A lot of modern development is wiring together large, specialized libraries. Our job is to take these things (along with their cascade of dependencies) and put our own little bits of glue and logic around them. And, heck, if I wanted to glue badly documented black boxes together, I would have gone into Electrical Engineering. But here's the thing. While the layers are getting thicker, the abstractions more opaque, in the end, much like a CEO is responsible for that ONE PERSON down in the factory behaving badly, we, as users of this stuff, are responsible for ALL OF IT. Down to bugs in the microprocessor. When push comes to shove, it's all on us. We can whine and complain and delegate. "I didn't write the compiler, not my fault." "Not my library..." "The Manager assured the VP who assured me that..." But doesn't really matter when you have a smoking crater of a system, does it? Because we're the ones delivering services and such, putting our names on it. So, yea, no, you don't have to be "as skilled", perhaps, when using something. But you're still responsible for it.
- ghuntley 1y ago
  Absolutely! I kind of hate the term "vibe coding" because of its associations with brain off. It is so important for an engineer to take accountability for what they ship. Now to your ponderoo about libraries, something I've found that's really fascinating is I've really stopped using open source libraries unless there's a network ecosystem effect for example like Tailwind. But for everything else, it's much easier to code generate it. And if there's something wrong with the implementation, I can take ownership and accountability for that implementation and I can just fix it with a couple more prompts. No more bullshit in open source. Some person might not even be maintaining the project anymore or having to deal with like getting a pull request fixed or open source supply chain attack vectors related to project takeovers and all that noise. It just doesn't exist anymore. It's really changed how I do software development.

jbellis 1y ago

Yes! This is an underappreciated point that both sides in "AI makes coding better" vs "AI just writes slop" have mostly missed. Co-authoring code with AI is a qualitatively different activity than writing code by hand, and it requires new skills. I started Brokk to give humans better tools with which to do this: https://github.com/BrokkAi/brokk https://github.com/BrokkAi/brokk

ghuntley 1y ago
Hey dude, it's been a couple weeks since we caught up for a zoom. Maybe it's three weeks now. Still keen to catch up again, dude. It's gonna be a little bit busy for the next couple of weeks. I've got two conference talks to do then I'm gonna be over in San Fran, but keen.

elliotbnvl 1y ago

> If they waste time by not using the debugger, adding debug log statements, or writing tests, then they're not a good fit. Do you mean writing manual tests? Because having the LLM write tests is key in iteration speed w/o backtracking.

ghuntley 1y ago
Typo on my part. I completely agree. Back pressure is everything. Edit: I've just updated the post.
- elliotbnvl 1y ago
  Nice. FWIW, since I just realized my only main comment was a criticism, I found your article very insightful. It baffles me how many people will disagree with the general premise or nit-pick one tiny detail. The only thing more surprising to me than the rate at which AI is developing is the number of developers jamming their heads into the sand over it.
  - ghuntley 1y ago
    Ah, I wrote a blog post specifically about this. Some engineers are not going to make it. https://ghuntley.com/ngmi https://ghuntley.com/ngmi
    - elliotbnvl 1y ago
      Oh, I read this the other day on HN and agreed strongly. I hadn't realized this was the same blog. Nice!
foldr 1y ago
The scope of the ‘not’ is also unclear here. I think they’re saying that you waste time if you don’t do all the things in the list, but it could also be read as saying that not using the debugger is a waste of time, and that adding debug log statements and writing tests is also a waste of time.
1y ago

bgwalter 1y ago

"Someone can be highly experienced as a software engineer in 2024, but that does not mean they're skilled as a software engineer in 2025, now that AI is here." With that arrogance, my only question is: Where is your own code and what makes you more qualified than Linux kernel or gcc developers?

elliotbnvl 1y ago
It's not arrogance, since they're not asserting anything about themself. It's a factual observation with which you're free to disagree – but should be challenged directly rather than via ad hominem if you want to actually make a point rather than just collect internet snark kudos. Also, there are far more generic web developers than there are Linux kernel developers, and they represent the vast majority of the market share / profit generation in software development, so your metric isn't really relevant either.
- skydhash 1y ago
  So what has changed about the realm of programming that make all the skill obsolete, including the skill to learn new programming thingies? The DOM API is old, All the mainstream backend languages are old, unix administrations has barely changed (only the way to use those tools have). Even Elasticsearch is 15 years old. Amazon S3 is past drinking age in many countries around the world. And that's just pertaining to web projects. You just need to open a university textbook to realize how old many of the fundamentals are. Most shiny new things is old stuff repackaged.
  - elliotbnvl 1y ago
    A lot of people are rejecting AI because of how transformational it is. Those people will fall behind the people who adopt it aggressively. It's akin to people who refused to learn C because they knew assembly.
    - skydhash 1y ago
      I don't think people has refused to learn C (which is not particularly hard to learn for someone who knows about assembly and various other languages at the time). A lot of compilers were buggy and people have lots of assembly snippets for particular routines. And that's not counting mature projects that was already in assembly and you have to maintain. A lot of programmers are actually fine trying new stuff out, but some are professionals and don't bring everything under the sun in their work projects.

brcmthrowaway 1y ago

Is there a good guide resource on properly using LLMs/agents for coding? How to get started?

ghuntley 1y ago
See the tweet on workflow that I put in the post. No courseware, no bullshit, it's there. Have fun. The blog has plenty of guidance from using specs to creating standard libraries of prompts and how to clone a venture capital-backed company while you sleep.
spacebanana7 1y ago
I found that learning prompt engineering was largely a waste of time. The value of the knowledge seems to depreciate so quickly. I spent loads of time learning to use special syntax that helped GPT 3.5 or comfy ui for Stable diffusion. Now the latest models can do exactly what I want without any of those “high skill” prompts. The context windows are so big that we can be quite lazy about dumping files into prompts without optimisation. The only general advice I’d give it to take more risk and continually ask more of the models.

mromanuk 1y ago

Every time I ask an LLM to write some UI and model for SwiftUI I have to specify to use @Observable macro (is the new way), which they normally do, after asking for it. The LLM tells me that they prefere the "older way" because it's more broadly compatible, that's ok if you are aiming for that. But If the programmer doesn't know about that they will be stuck with the LLM calling the shots for them.

maxwell 1y ago
It's just a higher level abstraction, subject to leaks as with all abstractions. How many professional programmers don't have assemblers/compilers/interpreters "calling the shots" on arbitrary implementation details outside the problem domain?
- LorenPechtel 1y ago
  But we trust those tools to do the job correctly. The compiler has considerable latitude in messing with the details so long as the result is guaranteed to match what was ordered--when we find any deviation from that even in an edge case we consider it a bug (Borland Pascal debugger, I'm looking at you. I wasted a *lot* of time on the fact that in single step mode you peacefully "execute" an invalid segment register load!) LLMs lack this guarantee.
  - maxwell 1y ago
    We trust those tools to do the job correctly now. https://vivekhaldar.com/articles/when-compilers-were-the--ai--that-scared-programmers https://vivekhaldar.com/articles/when-compilers-were-the--ai...
cpinto 1y ago
Have you tried writing rules for how you want things done, instead of repeating the same things every time?
starlust2 1y ago
A thing people miss is that there are many different right ways to solve a problem. A legacy system might need the compatibility or it might be a greenfield. If you leave a technical requirement out of the prompt you are letting the LLM decide. Maybe that will agree with your nuanced view of things, but maybe not. We're not yet at a point where LLM coders will learn all your idiosyncrasies automatically, but those feedback loops are well within our technical ability. LLM's are roughly a knowledgeable but naïve junior dev; you must train them! Hint: add that requirement to your system/app prompt and be done with it.

namuol 1y ago

> If I were interviewing a candidate now, the first things I'd ask them to explain would be the fundamentals of how the Model Context Protocol works and how to build an agent. Um, what? This is the sort of knowledge you definitely do not need in your back pocket. It’s literally the perfect kind of question for AI to answer. Also this is such a moving target that I suspect most hiring processes change at a slower pace.

dgfitz 1y ago
I would just give some sort of LLM-esque answer that sounds correct but is very wrong, and hope I would get the opportunity to follow that up with: "Oh, I must have hallucinated that, can you give me a better prompt?" I crack myself up.
- layer8 1y ago
  Don’t forget to apologize and acknowledge that they are right.
  - a_e_k 1y ago
    You're right to ask about that! ...
elliotbnvl 1y ago
It's a good heuristic for determining how read in somebody is to the current AI space, which is super important right now, regardless of being a moving target. The actual understanding of MCP is less important than the mindset having such an understanding represents.
- namuol 1y ago
  Hard disagree. It’s not super important to be AI-pilled. You just need to be a good communicator. The tooling is a moving target, but so long as you can explain what you need well and can identify confusion or hallucination, you’ll be effective with them.

foldr 1y ago

Isn’t this just a roundabout way of saying that people who are skilled at using LLMs will get better results from them than people who aren’t? In other words, LLMs “mirror operator skill” in about the same way as hammers, paintbrushes or any other tool. Hardly a blinding insight.

energy123 1y ago
It's a controversial opinion because both AI optimists and AI pessimists can find room for disagreement with the premise. The optimists think vibe coding is about to be fully automated and humans don't have long, one or two years at best. The pessimists think LLMs don't add much value in the first place. In either case they would disagree with the premise.
- marinmania 1y ago
  Agreed. By HN standards I am a very shitty programmer, and as of a year ago I would have said it takes up about 25% of my time. I pretty much just make demos to display some non-coding research. I think with the rise of LLMs, my coding time has been cut down by almost half. And I definitely need to bring in help less often. In that sense it has raised my floor, while making the people above me (not necessarily super coders, but still more advanced) less needed.

makmanalp 1y ago

Counterthoughts: a) These skills fit on a double sided sheet of paper (e.g. the claude code best practices doc) and b) what these skills are has been changing so rapidly that even the best practices docs fall out of date super quick. For example, managing the context window has become less of a problem with increased context windows in newer models and tools like the auto-resummarization / context window refresh in claude code make it so that you might be just fine without doing anything yourself. All this to say that the idea that you're left significantly behind if you aren't training yourself on this feels bogus (I say this as a person who /does/ use these tools daily). It should take any programmer not more than a few hours to learn these skills from scratch, with the help of a doc, meaning any employee you hire should be able to pick these up no problem. I'm not sure it makes sense as a hiring filter. Perhaps in the future this will change. But right now these tools are built more like user friendly appliances - more like a cellphone or a toaster than a technology to wrap your head around, like a compiler or a database.

patrickhogan1 1y ago

I agree with most of what you’re saying—especially the Unix pipe analogy and the value of building a prompt library while understanding which LLM to use. That said, I think there’s value in a catch-all fallback: running a prompt without all the usual rules or assumptions. Sometimes, a simple prompt on the latest model just works—and often in surprisingly more effective ways than one with a complex prompt.

henning 1y ago

Centering an interview around MCP and "building an agent" which often means "write a prompt and make HTTP calls to some LLM host service" is incredibly stupid unless that is what the product the company produces is. MCP is a tool and may only be minor in terms of relevance to a position. If I use tools that use MCP but our actual business is about something else, the interview should be about what the company actually does. Your arrogance and presumptions about the future don't make you look smart when you are so likely to be wrong. Enumerating enough predictions until one of them is right is not prescient, it's bullshit.

jmsdnns 1y ago

A key thing that has been shown in research at Wharton is that LLMs elevate people with less experience a lot more than they elevate people a lots of experience. If we take "operator skill" to mean "they know how to make prompts", there is some truth to it and we can see that by whether or not the operator is designing the context window or not. But for the more important question, whether or not LLMs are useful has an inverse relationship with how skilled the person is in the context they are using for. This is why the best engineers mostly shrug at LLMs while those that arent the best feel a big lift. So, LLMs are not mirrors of operator skill. This post is instead an argument that everyone should become prompt engineers.

ghuntley 1y ago
Keen to read the research. Can you drop the link?
- jmsdnns 1y ago
  I was at Penn's first AI conference last year and heard Dr Lilach Mollick's keynote where she said this is shown to be true over and over. She doesnt seem to publish often, but her husband Ethan always has a lot to say about AI. https://www.oneusefulthing.org/p/everyone-is-above-average https://www.oneusefulthing.org/p/everyone-is-above-average
  - ghuntley 1y ago
    Thanks.
namuol 1y ago
Disagree. Poor engineers will go in circles with AI because they will under specify their tasks and fail to recognize improper solutions. Ultimately if you’re not thoughtful about your problem and critical about the solution you will fail. This is true with or without AI at the wheel.
- Jensson 1y ago

deleted 1y ago

[deleted]

deleted 1y ago

[deleted]

satisfice 1y ago

A lot of these questions no one knows the answer to. If you think you know “the best LLM to use for summarizing” then you must have done a whole lot of expensive testing— but you didn’t, did you? At best you saw a comment on HN and you believed it. And if you did do such testing, I hope it wasn’t more than a month ago, because it’s out of date, now. The nature of my job affords me the luxury of playing with AI tech to do a lot of things, including helping me write code. But I’m not able to answer detailed technical questions about which LLM is best for what. There is no reliable and durable data. The situation itself is changing too quickly to track unless you have the resources to have full subscriptions to everything and you don’t have any real work to do.

incomingpain 1y ago

I have a greater appreciation of LLMs after working out how to do them offline. My 2 year old graphics card sure could be bigger... kicks can... it was plenty for starfield... holy crap vram is expensive! >If they waste time by not using the debugger, not adding debug log statements, or failing to write tests, then they're not a good fit. Why would you deprive yourself of a tool that will make you better?