13 ms·

Counterexample: Ive been able to complete more side projects in the last month leveraging llms than i have ever in my life. One of which I believe to have poten

• mythrwy 2y ago
  I've had both experiences strangely enough.
• talldayo 2y ago
  > which involved complicated rust `no_std` and linker setup for compiling rust code onto bare metal RISCV from scratch. That's complicated, but I wouldn't say the resulting software is complex. You gave an LLM a repetitive, translation-based job, and you got good results back. I can also believe that an LLM could write up a dopey SAAS in half the time it would take a human to do the same. But having the right parameters only takes you so far. Once you click generate, you are trusting that the model has some familiarity with your problem and can guide you without needing assistance. Most people I've seen rely entirely on linting and runtime errors to debug AI code, not "solid fundamentals" that can fact-check a problem they needed ChatGPT to solve first place. And the "experience" required to iterate and deploy AI-generated code basically boils down to your copy-and-paste skills. I like my UNIX knowledge, but it's not a big enough gate to keep out ChatGPT Andy and his cohort of enthusiastic morons. We're going to see thousands of AI-assisted success stories come out of this. But we already had those "pennies on the dollar" success stories from hiring underpaid workers out of India and Pakistan. AI will not solve the unsolved problems of our industry and in many ways it will exacerbate the preexisting issues.
  • HPsquared 2y ago
    What it will do is to free up a lot of brainpower to think about those hard problems and empower people to try our their ideas.
    • talldayo 2y ago
      I used to think the exact same thing would happen when we paid Pakistani and Indian labor to do America's busywork. That was about 15 years ago, I no longer have the same enthusiasm you do.
      • tonyhart7 2y ago
        Now, you are paying a Taiwanese or American company to produce GPUs for you. This allows you to use open-source models like DeepSeek R1, significantly reducing your reliance on Indian tech labor
        • sgc 2y ago
          I believe they are saying we did not learn to think more deeply then, so why should we expect to learn how now.
  • simonw 2y ago
    A tool that can "write up a dopey SAAS in half the time it would take a human to do" is a pretty incredible thing to add to your toolbox!
    • talldayo 2y ago
      If the summary goal of your existence is to be the most delirious waste of resources that humanity has yet known, sure. It's the hammer and nail of spoiled burnouts everywhere that need a credible ruse to help them out of the bottle. Some of us are capable of wanting for things better than a coin-operated REST API. The kind of imagination used to put people on the moon, that now helps today's business leaders imagine more profitable ways to sell anime pornography on iPhone. (Don't worry, AI will disrupt that industry too.)
• baxtr 2y ago
  Is it reasonable to assume that more senior devs benefit more from LLMs?
  • dogma1138 2y ago
    It depends it think it’s less about how senior they are and how good they are at writing requirements, and knowing what directives should be explicitly stated and what can be safely inferred. Basically if they are good at utilizing junior developers and interns or apprentices they probably will do well with an LLM assistant.
  • adamtaylor_13 2y ago
    I believe so. In my experience, you need to have that gut intuition (or experience) to say, “No way. That’s totally wrong.” Since AI will capitulate and give you whatever you want. You also have to learn how to ask without suggesting because it will take whatever you give it and agree.
    • j_bum 2y ago
      Yep. I think a default state of skepticism is an absolute necessity when working with these tools. I love LLMs. I agree with OP them expanding my hobby capacity as well. But I am constantly saying (in effect) “you sure…?” and tend to have a pretty good bs meter. I’m still working to get my partner to that stage. They’re a little too happy to accept an answer without pushback or skepticism. I think being ‘eager to accept an answer’ is the default mode of most people anyway. These tools are likely enabling faster disinformation consumption for the unaware.
  • jumpman500 2y ago
    Ya. I think people that have better technical vocabulary and an understanding of what should be possible with code do better. That’s usually a senior engineer, but not always
  • RickS 2y ago
    IMO experience provides better immunity for common hangups. Generated code tends to be directionally pretty good, but with lots of minor/esoteric failures. The experience to spot those fast and tidy them makes all the difference. Copilot helps me move 10x faster with tedious arduino stuff, but I can easily see where if I didn't have decent intuition around debugging and troubleshooting, there'd be almost zero traction since it'd be hard to clear that last 10% hurdle needed to even run the thing.
• dogma1138 2y ago
  Indeed LLMs are useful as an intern, they are at the “cocky grad” stage of their careers. If you don’t understand the problem and can’t steer the solution and worse has only limited understanding of the code they produce you are unlikely to be productive. On the other hand if you understand what needs to be done, and how to direct the work the productivity boost can be massive. Claude 3.5 sonnet and O1 are awesome at code generation even with relatively complex tasks and they have a long enough context and attention windows that the code they produce even on relatively large projects can be consistent. I also found a useful method of using LLMs to “summarize” code in an instructive manner which can be used for future prompts. For example summarizing a large base class that may be reused in multiple other classes can be more effective than having to overload a large part of your context window with a bunch do code.
• deleted 2y ago
  [deleted]
• pfannkuchen 2y ago
  I’ve found LLMs to basically be a more fluent but also more lossy way of interfacing with stack overflow and tutorials. If a topic is well represented in those places, then you will get your answer quicker and it can be to some extent shaped to your use case. If the topic is not well represented there, then you will get circular nonsense. You can say “obviously, that’s the training data”, and that’s true, and I do find it obvious personally, but the reaction to LLMs as some kind of second coming does not align with this reality.
  • biohcacker84 2y ago
    That matches my experience too. I wonder how fast they'll improve and if LLMs will hit a wall, as some AI experts think.
    • janderson215 2y ago
      Is it possible that you’re both using LLMs the same way you’d use SO and that’s the reason you see such similarities? The reason I ask is because it doesn’t not match my experience. It feels more like I’m able to Matrix-upload docs into my brain like Trinity learning to fly a helicopter.
      • h0l0cube 2y ago
        > I’m able to Matrix-upload docs into my brain like Trinity learning to fly a helicopter. So you're using it like Wikipedia? I find when learning something new (and non coding related) YouTube is infinitely better than an LLM. But then I prefer visual demonstration to tutorial or verbal explanation.
      • pfannkuchen 2y ago
        I am using it like stack overflow in the sense that I’m solving a problem and I’m using it to answer questions when I’m in an unfamiliar or non-obvious place in the problem space. If I have a question about a first order language or framework feature or pattern, it works great. If I have a question about a second order problem, like an interaction between language or framework features, or a logical inconsistency in feature behavior, then it usually has no idea what’s going on, unless it turns out to be a really common problem such as something that would come up when working through a tutorial. For code completion, I’ve just turned it off. It saves time on boilerplate typing for sure, but the actual content pieces are so consistently wrong that on balance I find it distracting. Maybe I have a weird programming style that doesn’t mesh well with the broader code training corpus, not sure. Or maybe a lot of people spend more time in the part of problem-space that intersects with tutorial-space? I am not very junior these days. That being said I definitely do use LLMs to engage with tutorial type content. For that it is useful. And outside of software it is quite a bit better for interfacing with Wikipedia type content. Except for the part where it lies to your face. But it will get better! Extrapolating never hurt anyone.
• maybesomaybenot 2y ago
  Claude is like having my own college professor. I've learned more in the past month with Claude then I learned in the past year. I can ask questions repeatedly and get clarification as fine as a need it. Granted, Claude has limits, but its a game-changer. > I think the key to being successful here is to realize that you're still at the wheel as an engineer. The llm is there to rapidly synthesize the universe of information. Bingo. OP is like someone who is complaining about the tools, when they should be working on their talent. I have a LOT of hobbies (circuits, woodworking, surfing, playing live music, cycling, photography) and there will always be people who buy the best gear and complain that the gear sucks. (NOTE: I"m not implying claude is "the best gear", but it's a big big help.) I think the only problem with LLMs is synthesis of new knowledge is severely limited. They are great at explaining things others have explained, but suck hard at inventing new things. At least that's my experience with Claude: it's terrible as a "greenfield" dev.
  • yousoundawesome 2y ago
    [flagged]
  • XenophileJKO 2y ago
    I would add though, that they can be very good at combining known concepts. Which can create a non-trivial set of "new knowledge".
    • maybesomaybenot 2y ago
      Creating new knowledge from current knowledge is called "synthesis" (ancient term, nothing modern). I'm hoping you're right, it would be amazing.
  • KerrAvon 2y ago
    How do you know it's accurate?
    • selcuka 2y ago
      They are reasonably accurate, and no tutor is perfect. How do you know your college professor is accurate?
• EarthMephit 2y ago
  I find that LLMs are almost comically bad at projects that have a hardware component like RaspberryPi or Pico, or Ardunio. I think that its because often the libraries you use are niche or have a a few similar versions, the LLM really commonly hallucinated solutions and would continually suggest that library X did have that capability. I think because often in hardware projects you often hit a point where you can't do something or you need to modify a library, but the LLM tries to be "helpful" and it makes up a solution.
  • chamomeal 2y ago
    They’re also pretty bad at typescript generics. They’re quite good at explaining concepts (like mapped types), but when push comes to shove they generate all sorts of stuff that looks convincing, but doesn’t pass the type checker. And then you’ll paste in the error, and they’ll just say “ok I see the problem” and output the exact same broken code lol. I’m guessing the problem is lack of training data. Most TS codebases are mostly just JS with a few types and zod schemas. All of the neat generic stuff happens in libraries or a few utilities
  • jerf 2y ago
    Based on my own modestly successful forays into that world, I have to imagine one problem the LLMs have in that space is terrible training data. A good three quarters of any search result you search for in that space will be straight-up out-of-date and not work on your system. Then you've got all the tiny variations between dozens of chipsets, and all the confidently wrong people on the internet telling you to do nonsensical things, entire ecosystems that basically poofed out of existence three years ago but are still full of all kinds of juicy search terms and Google juice... if I can hardly paw through this stuff with years of experience and the physical hardware right in front of me to verify claims with, I don't know how an LLM is supposed to paw through all that and produce a valid answer to any question in that space, without its own hardware to fiddle with directly.
    • lostlogin 2y ago
      You’re making my eye twitch. The number of projects I’ve done where my notes are the difference between hours of relearning The Way or instant success. Google doesn’t work as some niche issue is blocking the path. ESP32, Arduino, Home Assistant And various media server things.
• onemoresoop 2y ago
  I find LLMs to be decent unblockers. I only turn to them form time to time though, unless Im in a playful mode and try poking out various ideas. As a coder I also ask for snippets when Im lazy. I tried growing a slightly larger solution a few times and it failed in dumb ways. It was clear it doesn’t really comprehend we do, it’s not aware it’s moving in circles and so on. All these things will probably see a lot incremental of improvents and as a tool will definitely stay but fundamentally LLMs can’t really think, at least the way we do and expecting that is also foolish.
• gerdesj 2y ago
  [flagged]
  • potsandpans 2y ago
    You are very smart and cool.
    • gerdesj 2y ago
      "You are very smart and cool." No I am really not. You aren't either.
• chamomeal 2y ago
  I agree with his point about asking AI to “fix” problems though. It’s really nice that you don’t have to fully understand something to use it, but that becomes a problem if you lean on it too much
• gsf_emergency_2 2y ago
  Ime engineers who find LLM useful have misunderstood their reasons for existence outside of being salaried.. What is your main project ? Do you LLM that? ( I wager you're not a rust expert and should maybe reconsider using rust in your main project. FWIW asking LLM whether you should use rust ~ asking it about the meaning of life. Important questions that need answers, but not right away! (A week or 2 tops) If you need to synthesize the universe of information with LLM.. that is not the universe you want to live or play in
• blast 2y ago
  > The llm is there to rapidly synthesize the universe of information That's a nice way of putting it.
• pinoy420 2y ago
  Exactly this. OP, credit where credit is due, appears to be someone who “hacks things together” copy pasting solutions blindly from the internet - with little intuition gained along the way.
• irthomasthomas 2y ago
  Another data point. Plugin ideation to publication in 2 minutes, from a single prompt, albeit a multi-step shell prompt. 3 packages published in 24 hrs https://x.com/xundecidability/status/1884077427871342955 https://x.com/xundecidability/status/1884077427871342955
• zackproser 2y ago
  This