7 ms·

> LLMs are very useful tools for software development That's an opinion many disagree with. As a matter of fact, the only limited study up to date showed that

• yakshaving_jgt 1y ago
  I’m 15 years into my career and I write Haskell every day. I’m getting a massive productivity boost from using an LLM.
  • deleted 1y ago
    [deleted]
  • black_knight 1y ago
    How do you find the quality of the Haskell code produced by LLM? Also, how do you use the LLM when coding Haskell? Generating single functions or more?
    • yakshaving_jgt 1y ago
      I'm stuck in my ways with vim/tmux/ghci etc, so I'm not using some AI IDE. I write stuff into ChatGPT and use the output, copying manually, or writing it myself with inspiration from what I get. I feed it a fair bit of context (like, say, a production module with a load of database queries, and the associated spec module) so that it copies the structure and patterns that I've established. The quality of the Haskell code is about as good as I would have written myself, though I think it falls for primitive obsession more than I would. Still, I can add those abstractions myself after the fact. Maybe one of the reasons I'm getting good results is because the LLM effectively has to argue with GHC, and GHC always wins here. I've found that it's a superpower also for finding logic bugs that I've missed, and for writing SQL queries (which I was never that good at).
      • meowface 1y ago
        Try Claude Code.
        • yakshaving_jgt 1y ago
          Why?
          • tommyengstrom 1y ago
            I use similar style as you. neovim with ghci inside, plus hls, and ghciwatch. Claude code is nice because it is just a separate cli tool that doesn't force you to change editor etc. It can also research things for you, make plans that you can iterate before letting it loose, etc. Claude is also better than chatgpt at writing haskell in my experience.
            • wahnfrieden 1y ago
              Codex CLI with gpt-5-thinking on "high" reasoning is also good to try as an alternative now
      • black_knight 1y ago
        “GHC always wins” is a nice sentiment. Another similar thing happens when I have written QuickCheck tests and get the LLM to make the implementation conform. Quickcheck almost always wins that fight as well.
    • tommyengstrom 1y ago
      I'm in a similar situation. I write Haskell daily and have been working with Haskell for a bunch of years. Though I use claude code. The setup is mostly stock, though I do have a hook that feeds the output of `ghciwatch` back into claude directly after editing. I think this helps. - I find the code quality to be so-so. It is much more into if-then-else than the style is to yolo for my liking. - I don't rely on it for making architectural decisions. We do discuss when I'm unsure though. - I do not use it for critical things such as data migrations. I find that the errors is makes are easy to miss, but not something I do myself. - I let it build "leaves" that are not so sensitive more freely. - If you define the tasks well with types then it works faily well. - cluade is very prone to writing tests that test nothing. Last week it wrote a test that put 3 tuples with strings in a list and checked the length of the list and that none of the strings where empty. A slight overfit on untyped languages :) - In my experience, the uplift from Opus vs Sonnet is much larger when doing Haskell than JS/Python. - It matters a lot if the project is well structured. - I think there is plenty of room to improve with better setup, even without models changing.
• manmademagic 1y ago
  I wouldn't call myself an 'experienced' developer, but I do find LLMs useful for once-off things, where I can't justify the effort to research and implement my own solution. Two recent examples come to mind: 1. Converting exported data into a suitable import format based on a known schema 2. Creating syntax highlighting rules for language not natively support in a Typst report Both situations didn't have an existing solution, and while the outputs were not exactly correct, they only needed minor adjustments. Any other situation, I'd generally prefer to learn how to do the thing, since understanding how to do something can sometimes be as important as the result.
• wahnfrieden 1y ago
  That's a skill issue. That lone study was observing untrained participants. It's no surprise to me that devs who are accustomed to working on one thing at a time due to fast feedback loops have not learned to adapt to paralellizing their work (something that has been demonized at agile style organizations) and sit and wait on agents and start watching YouTube instead, as the study found (productivity hits were due to the participants looking at fun non-work stuff instead of attempting to parallelize any work). The study reflects usage of emergent tools without training, and with regressive training on previous generation sequential processes, so I would expect these results. If there is any merit in coordinating multiple agents on slower feedback work, this study would not find it.
  • ath3nd 1y ago
    Interesting take. I suggest an alternative take: it's a skill issue if LLMs help a developer. If the study showed that experienced developers suffered a negative performance impact while using an LLM, maybe where LLMs shine are with junior developers? Until a new study that shows otherwise comes out, it seems the scientific conclusion is that junior developers, the ones with the skill issues, benefit from using LLMs, while more experienced developers are impacted negatively. I look forward to any new studies that disprove that, but for now it seems settled. So you were right, might indeed be a skills issue if LLMs help a developer and if they do, it might be the dev is early in their career. Do LLMs help you, out of curiosity?
    • sokoloff 1y ago
      Imagine if you’d worked for a decade as a dev using Notepad as your code editor (in a world where that was the best editor somehow). You’d developed your whole career in Notepad and knew very well how to work with it Then, someone did a two week study on the productivity difference between Notepad, vim, emacs, and VSCode. And it turns out that there was lower observed productivity for all of the latter 3, with the smallest reduction seen in VSCode. Would you conclude that Notepad was the best editor, followed by VSCode and then vim and emacs being the worst editors for programming? That’s the flaw I see in the methodology of that study. I’m glad they did it, but the amount of “Haha, I knew it all along and if you claim AI helps you at all, it’s just because you sucked all along…” citing of that study is astonishing.
• ardit33 1y ago
  LLMs help a lot in doing 'well defined' tasks, and things that you already know you want, and they just accelerate the development of it. You still have to re-write some of it, but they do the boring stuff fast. They are not great if your tasks are not well defined. Sometimes, they suprise you with great solutions, sometimes they produce mess that just wastes your time and deviates from your mission. To, me LLMs have been great accelerants when you know what you want, and can define it well. Otherwise, they can waste your time by creating a lot of code slop, that you will have to re-write anyways. One huge positive sideffect, is that sometimes, when you create a component, (i.e. UI, feature, etc), often you need a setup to test, view controllers, data, which is very boring and annoying / time wasting to deal. LLM can do that for you within seconds (even creating mock data), and since this is mostly test code, it doesn't matter if the code quality is not great, it just matters to get something in the screen to test the real functionality. AI/LLMs have been a huge time savers for this part.
  • Terr_ 1y ago
    I get the impression that the software scenarios where LLMs do the best on both reliability and time-saving are places where a task was already ripe (or overdue) to be be abstracted away: Turned into a reusable library; as as a default implementation or setting; expressed as a shorter DSL; or a template/generator script. When it's a problem lots of people banged their head against and wrote posts about similar solutions, that makes for good document-prediction. But maybe we should've just... removed the pain-point.
• 9rx 1y ago
  > decrease productivity for experienced developers by roughly 19%. Seems about right when trying to tell an LLM what to code. But flipping the script, letting the LLM tell you what to code, productivity gains seem much greater. Like most programmers will tell you: Writing code isn't the part of software development that is the bottleneck.
• CuriouslyC 1y ago
  People who suck at typing are better off writing by hand as well. I don't need to argue, I'll let history pick a winner.
  • ath3nd 1y ago
    People who get overly excited from every new shining thing also thought that NFTs and Crypto and web3 (whatever the heck it means) are the next coming of Jesus. If LLM boosters were not so preachy about it, I'd left them off the hook easier. But at the current moment: - Only study up to date shows experienced developers have 19% less productivity when using LLMs https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-stud https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o... - There are studies showing that using LLMs regularly makes you dumber https://www.media.mit.edu/publications/your-brain-on-chatgpt/ https://www.media.mit.edu/publications/your-brain-on-chatgpt... - The fresh study from MIT shows 95% of AI pilots fail https://fortune.com/2025/08/18/mit-report-95-percent-generative-ai-pilots-at-companies-failing-cfo/ https://fortune.com/2025/08/18/mit-report-95-percent-generat... - The companies developing LLMs don't have that part of their business profitable nor any path for profitability. You can see it with Anthropic's constantly changing token limits and plans, and with Microsoft and OpenAI not able to reach a deal https://www.ft.com/content/b81d5fb6-26e9-417a-a0cc-6b6689b70c98 https://www.ft.com/content/b81d5fb6-26e9-417a-a0cc-6b6689b70... - Hell, Sam Altman himself admitted that the current Ai market is just a bubble https://www.cnbc.com/2025/08/18/openai-sam-altman-warns-ai-market-is-in-a-bubble.html https://www.cnbc.com/2025/08/18/openai-sam-altman-warns-ai-m... When the LLM cultists wake up during the bubble pop, I wonder what they are gonna jump on next. The world is running out of hype bandwagons to jump on. Maybe... LLM NFTs?
    • CuriouslyC 1y ago
      If we put people in a jet with poor training and they crash, that's the pilot's fault, yet if people crash LLMs, that's the LLMs fault. If a study showed 95% of people crashed jets without training, I wouldn't take that as a sign jets are a flawed idea. As it is, I have no problem with your naysaying, I'm getting results, your disbelief doesn't change that, in fact I find it more amusing than anything.
• jopsen 1y ago
  There are many ways to use an LLM. Writing code is a bit crazy, maybe writing tedious test case variations. But asking an LLM questions about a well established domain you're not expert in is a fantastic use case. And very relevant for making software. In practice, most software requires you to understand the domain your aiming to serve.
  • ChrisMarshallNY 1y ago
    Not sure why your post was dinged. I have used them for exactly this, and it has been amazingly effective.
• ChrisMarshallNY 1y ago
  Probably depends on how it’s being used. I use LLMs every day. They are useful to me (quite useful), but I don’t really use them for coding. I use them as a “fast reference” source, an editor, or as a teacher. I’ve been at this since 1983, so I’ve weathered a couple of sea changes.
  • godelski 1y ago
    My favorite use is LLMs is a fuzzy search. Give them a description, to search that, iterate. Or get them to role play an expert in some field. Doesn't matter if they hallucinate. Take that jargon and use it to improve your searches. They're super helpful in these contexts. But these are also contexts where I don't need to rely on accuracy.
• ftmootnomoat 1y ago
  This kind of blanket statement smells of the same dogmatism as the AI hype train in reverse. LLMs are a just a simple tool, if people misuse it it's on them.
• antonvs 1y ago
  > showed that LLMs usage decrease productivity for experienced developers by roughly 19%. That’s a massive overstatement of what the study found. One big caveat is this: “our developers typically only use Cursor for a few dozen hours before and during the study.” In other words, the 19% slowdown could simply be a learning curve effect. > one has to be really early in their career to benefit from their usage. I have decades of experience, and find them very beneficial. But as with any tool, it helps to understand what they are and aren’t good at, and hope to use them effectively. That knowledge comes with experience. Be careful of dismissing a new tool just because you haven’t figured out how to use it effectively.