3 ms·
I Program with LLMs
- taylodl 2y agoI've been using LLMs the same way this past year. As a challenge to myself, I've implemented full-stack interview evaluations using languages and tools I've never worked with before and see what I can do. Using LLMs, it hasn't taken me more than 50% longer to craft a solution as compared to my crafting one using tools and technologies with which I'm intimately familiar. Some of that extra time was a result of learning the craft of using LLMs in this manner. With each new technology stack, that excess amount of time decreased. Now, let's be honest, you still have to have excellent development skills, and it helps immensely to have a good knowledge of software architecture and design. I say that to point out that non-developers could not achieve these results. In the right hands though, it can make you a lot more productive.
- rafaelmn 2y ago> I've been using LLMs the same way this past year. As a challenge to myself, I've implemented full-stack interview evaluations using languages and tools I've never worked with before and see what I can do. Using LLMs, it hasn't taken me more than 50% longer to craft a solution as compared to my crafting one using tools and technologies with which I'm intimately familiar. The problem with this is that you don't know what you don't know. I've actually used Claude to build a project in a framework that I haven't worked with before and it got me pretty far. Once I got that reviewed by someone who actually knew the framework it was terribly structured, not using higher level constructs to organize things, wasn't using idiomatic approaches - stuff I could easily confirm myself after going through the official docs. I think using LLMs to skip learning from official sources is a bad move, you're bound to hit a wall with LLM (at the very least the training data cutoff could be before something changed in the framework), you'll have to deep dive into the docs and figure it out. That is to say - sure you can pass the interview with LLM only - but you'll suck at the job. If you invest some effort upfront to learn from traditional sources and then use LLM to speed you up it's going to pay off really fast IMO. The more I know about what the output should be the more useful LLMs are.
- Tostino 2y agoRight. I've had that same problem when trying to build a new app with Claude / 4o. I essentially have to go through the docs myself so I understand the framework structure enough to write up an outline for the llm to follow when developing a new component or backend structure.
- dagw 2y agoWith Claude I've had great success with first uploading the docs so that it has them as context for the rest of the session. Without doing that I got all kinds of answers, sometimes using a different framework or an older version of the framework I was using. Once it had the documentation to reference it started giving a lot better answers
- Bluestein 2y agoGreat point. Great suggestion.-
- cadamsdotcom 2y agoVery interesting outcome. Shows how far it’s possible to get without being idiomatic. You say the docs contain guidance on structuring, high-level constructs, and how to make things idiomatic. It would be an interesting test to hand the unfixed revision of the code to an LLM while also giving it the docs, and say “make any fixes to make this conform to standards of the framework and libraries”. If it picks up the same things that’s great news for novice programmers and anyone new to a framework! LLMs will improve at making these fixes over time - even if they’re currently bad at it that won’t last.
- rafaelmn 2y agoBut that's kind of my point - when I was originally working on it - I didn't know there's a more idiomatic approach - why would I copy paste the docs to ask it to improve the results ? Especially given the LLMs tendency to hallucinate and gnerate noise when asked to "improve code". Also without reading the docs would I be able to spot why the idiomatic approach is better from single use case ? You're basically depending on LLM to make choices for you - and in my experience so far it makes very suboptimal ones.
- nullc 2y agoIt works much better when you already know what you're doing-- so you can ask for exactly the right thing, spot the LLMs errors, and know what you need to do to fix them rather than iterating forever with the hallucinating machine. I can't help but feel that while these tools speed everyone up they actually increase the difference between an expert and a novice rather than leveling the playing field.
- bryanlarsen 2y agoSteve Yegge has laid this thesis out: https://sourcegraph.com/blog/the-death-of-the-junior-developer https://sourcegraph.com/blog/the-death-of-the-junior-develop... He also has some good content out there on how to program using LLM's. He calls it Chat Oriented Programming, aka CHOP. That's a good term to Google for.
- whstl 2y agoI enjoyed the examples. My previous company's lawyers really had success writing complex contracts in LLMs. The writer one really hits home: a friend who's a writer is able to make amazing stuff with ChatGPT, because he knows how to do it himself. I can only make stupid shit, because I while can recognise quality writing, I don't really know why. Similarly, I see the same thing happening with junior devs: a lot of them just can't get the answers I get out of Copilot or ChatGPT. I have to sit next to them and tell them what to ask the LLM, or when to say no to a proposed solution. Similarly, I can get started myself on Computer Vision and Machine Learning projects with LLMs, but I need to hand over to a proper CV or ML Engineer to get the last 10% done.
- lgas 2y ago> Similarly, I can get started myself on Computer Vision and Machine Learning projects with LLMs, but I need to hand over to a proper CV or ML Engineer to get the last 10% done. Unless by "Computer Vision and Machine Learning" projects, you mean you are trying to develop new approaches to computer vision or ML, I don't see why you would need to involve a CV/ML engineer? The stuff is all generally packaged as re-usable libraries like everything else, no?
- senko 2y agoReposted from original: https://crawshaw.io/blog/programming-with-llms https://crawshaw.io/blog/programming-with-llms HN discussion of original a few days ago: https://news.ycombinator.com/item?id=42617645 https://news.ycombinator.com/item?id=42617645
- blakesterz 2y agoJust a quick note that the title is "How I Program With LLMs", this is currently missing "How" which changes the meaning a bit, at least for me.
- nuancebydefault 2y agoIf i am not mistaken, HNs automated, somewhat pedantic filter/formatter at work. The rationale being, it tries to reduce any click-baity content by making titles a bit more serious/boring.
- nickfixit 2y agoMe as well. But sometimes I like the hallucinations of aider and other tools and just let it iterate over and over and over. Kind of fun to see. But by no means a drop in replacement for actual knowledge.
- xqzv 2y agoIs it too much to ask for a "How I use LLMs to program" article not come from someone who is actively marketing their AI startup?
- dingnuts 2y agoOr at least call it out as a disclaimer? I read this whole article a few days ago when it came by on this site and that would've been real nice to know. Really puts the whole thing in a different perspective, knowing he's FUCKING SELLING ME SOMETHING. I'm so sick of this!
- m_kos 2y agoYou have touched upon something important here. I have spent the last six months thinking about the role of disclaimers in how written text is perceived. Disclaimers, those seemingly innocuous statements meant to protect the author from liability or misinterpretation, often have the unintended consequence of undermining the credibility and impact of the work itself. By preemptively acknowledging potential flaws or limitations, authors can inadvertently signal a lack of confidence in their own arguments. This can lead readers to question the validity of the information presented, even if the disclaimer itself is minor or irrelevant to the core message. The very act of issuing a disclaimer can create a sense of distrust. It suggests that the author anticipates skepticism or potential backlash, raising questions about their motives and the integrity of their work. This is particularly problematic in academic settings, where disclaimers can undermine the authority and rigor of research findings. While acknowledging limitations is crucial for responsible scholarship, the way these limitations are framed can significantly impact how the research is received and interpreted. A carefully worded disclaimer can acknowledge potential weaknesses without undermining the overall strength of the argument, while a poorly constructed one can create unnecessary barriers to engagement and understanding. For this reason, I created Craft Your Disclaimer, an AI-powered service to help creators craft disclaimers tailored to their goals and audience preferences. You can register for free at: CraftYourDisclaimerOrSomeOtherBS.ai