3 ms·

My worry about this approach is: there is a reasonably popular saying that writing code is hard but debugging it is twice as hard (at least), which I think is a

by oblio 2y ago

My worry about this approach is: there is a reasonably popular saying that writing code is hard but debugging it is twice as hard (at least), which I think is an accurate description. LLMs will greatly increase code production, will they also increase debuggability to match?

TeMPOraL 2y ago
LLMs don't just make it easy to accumulate code - they make it easy to throw code away and start again. This already enables taking a different approach to debuggability - if there's a bug and it can't be trivially solved, trash that bit of the code and write it again. It may not be broadly viable just yet, but it will be if the models keep getting cheaper and better. This is also implied in the idea of "vibe coding" - don't bother understanding the code or debugging it yourself; if it doesn't work the way you like, just say it and have the model fix it until it gets it right (or you run out of money).
- Apocryphon 2y ago
  Sounds very maintainable and rigorously constructed
  - jryle70 2y ago
    How do you know it won't be maintainable? based on your current knowledge?
    - Apocryphon 2y ago
      Generating tons of throwaway code that is swapped out every time a bug occurs does not sound like engineering for the long-term.
      - TeMPOraL 2y ago
        Since when does any business care about "engineering for the long-term"? We, the software engineers, care. The business doesn't. In fact, the industry has systematically been trying to beat the care out of engineers - it's unprofessional to care about the work beyond the point it stops making money for the business. I'm not saying this is right or wrong - but this is how companies roll; if they can fix product issues by having AI throw chunks of code away and do it again, if that's reliably cheaper than having engineers do actual engineering, then that's what businesses will do.
        Apocryphon 2y ago
        What happens when this leads to high-profile disasters? https://news.ycombinator.com/item?id=43449642 https://news.ycombinator.com/item?id=43449642
        TeMPOraL 2y ago
        Hopefully it won't. You don't put webshit in control of rockets or cars, and you shouldn't put vibecoded software in control of them either. Programming safety-critical systems is its own thing, and it should be resistant to LLM incursions at least as long as human sign-off is an important part of the job.
- Jensson 2y ago
  > LLMs don't just make it easy to accumulate code - they make it easy to throw code away and start again. To throw away code you have to understand it, so no its the opposite. Code you don't understand is the hardest to get rid of, so it stays the longest in your codebase. > if there's a bug and it can't be trivially solved, trash that bit of the code and write it again How do you know where the bug is if you don't understand the code? There is no known algorithm to take a bug description and return the place in the code the bug is, otherwise bug fixing would be trivial. Edit: Not to mention that in real production systems your bugs will corrupt the database, and if you haven't set up a logging system etc you will likely not realize for a while forcing you to do a rollback to a very old state losing so much data. You wont last long doing that.
  - TeMPOraL 2y ago
    > To throw away code you have to understand it, so no its the opposite. Code you don't understand is the hardest to get rid of, so it stays the longest in your codebase. No, you don't. > How do you know where the bug is if you don't understand the code? There is no known algorithm to take a bug description and return the place in the code the bug is, otherwise bug fixing would be trivial. Yes, there is. At worst, the bug is somewhere in the entire project. But you probably have a more narrow idea where the bug is, or when it was introduced. "In module X", "In feature Y", "In the last N days/weeks". Not to mention, for most bugs, `git bisect` is enough to precisely narrow down the problematic change, and doing that doesn't actually require understanding anything about the code. It all boils down to costs. Even if it takes AI a whole day and 1000 attempts to do what would be a relatively simple fix for an experienced developer, if those 1000 attempts cost less than the developer's work-hours, the business will soon learn to prefer AI over people. If and when we get to that point, is mostly just a function of LLM performance and cost. If they get cheap enough, it'll make as much sense to have human developers fix bugs in code, as it makes sense for you to mend holes in your socks instead of buying them in bulk on-line. > Not to mention that in real production systems your bugs will corrupt the database, and if you haven't set up a logging system etc you will likely not realize for a while forcing you to do a rollback to a very old state losing so much data. This really depends on the kind of system you're doing, and the kind of data you're storing.
- Panzer04 2y ago
  Ever heard of chesterton's fence? If you don't understand what the code is doing, removing it sounds like a recipe for disaster.
  - TeMPOraL 2y ago
    Have you ever heard of shirts? In the past, if you'd tear your shirt, you'd spend time mending it, or pay someone to do it for you. Today, you just throw it in the trash and buy a new one, as it's much cheaper and faster. Think of any other goods we don't repair anymore. Regardless of their internal complexity and beauty of engineering, and no matter how small the defect is, if it's cheaper to replace it wholesale than to repair it, people end up replacing it. There's no reason to believe the same won't happen to code.
    - ModernMech 2y ago
      That's only cheaper because we've made it someone else's problem. Someone is paying that cost -- fast fashion is a real issue that causes waste, pollution, and human rights violations. Now as we face tarriffs on foreign manufactured goods, things don't seem so cheap anymore. Mending socks looks a little better. You're right, there's no reason to believe the same won't happen to code, but there's also no reason to believe it won't similarly end in all kinds of problems that come back to bite us down the line when the goodtimes are over.
- anon373839 2y ago
  > if there's a bug and it can't be trivially solved, trash that bit of the code and write it again When I was in undergrad, I knew a few guys who approached every problem by pasting in snippets from Stack Overflow and tutorial sites until the code “worked”. Did not end well…
  - TeMPOraL 2y ago
    That was then, and this is now. Look, I understand the sentiment. I too want the code to be done properly, well-engineered and thought through. But recall, this is not our job. We are Professionals, and by modern definition, a Professional does whatever is best for the business. And the business doesn't care about the product - it cares about the product's ability to earn them money. If generating shit code, then throwing it away and generating anew at any sign of bug (or spec change) gets cheap enough, this is what the business will want to do. This is what being Professional will mean. (You can imagine I don't hold "professionalism" in a very high regard.) See also: basic goods in meatspace. In developed economies, people generally don't repair clothes anymore, and increasingly rarely anyone bothers with repairing appliances. It's cheaper to just throw it away and buy a new one, than to try and repair it. Hell, even construction and remodeling these days involves a lot more of "affix it here permanently; if you need to move it, just smash it and install a new one" approach. Why wouldn't the same eventually happen with code?
    - deleted 2y ago
      [deleted]
    - anon373839 2y ago
      This is an interesting thought experiment, but I don’t think it’s realistic. Software that “works” doesn’t necessarily work. And software that doesn’t work can cost catastrophic financial losses. Worse yet, those losses are going to be felt more painfully when they are incurred recklessly. There is a reason that humans developed analytical problem solving as an alternative to trial and error: when you can do it, it’s more effective, and safer. That’s not to say that disposable code doesn’t have some interesting implications! One of them is that experimentation becomes a lot cheaper, so it’s faster to navigate the search space of possible solutions to a problem. But just taking a “solution” directly from an LLM without validating its correctness is fundamentally unserious and will be punished by reality sooner or later.
lukeschlather 2y ago
o1 has lately seemed more useful for debugging than for actually writing code. With writing code, it may give me some scaffold but I have a pretty particular idea of what I want and I have to rewrite most of it. With debugging, there have been multiple cases where I was stuck on something, I described the problem in detail and o1 gave me an insightful explanation of what I didn't understand. It's not magic, but mostly what makes it not magic is it can't respond to very detailed questions. But if I can get it the relevant information into a small space it can draw connections I can't.
cameldrv 2y ago
I think probably people will also use LLMs to debug. At this point, I think you can consider vibe-coding with an LLM to be pretty equivalent to using a fairly junior developer with access to stack overflow. It's going to make a lot of mistakes, it's going to make a lot of questionable decisions. Sometimes it will be able to fix its mistakes, sometimes it will spin its wheels and never fix it. It may make a big ball of mud. The entire project may fail. Two-three years from now, who knows. I'm pretty sure you still won't see 100% reliability in translating English->code, but you also don't see that even with senior developers.
- corysama 2y ago
  > I think probably people will also use LLMs to debug. I don’t have a link handy, but someone has already set up a service that’s “JS time travel debugger + LLM that knows how to use it” Pretty sure it was a Show HN recently.
  - istjohn 2y ago
    https://news.ycombinator.com/item?id=43023698 https://news.ycombinator.com/item?id=43023698

by oblio 2y ago

TeMPOraL 2y ago

LLMs don't just make it easy to accumulate code - they make it easy to throw code away and start again. This already enables taking a different approach to debuggability - if there's a bug and it can't be trivially solved, trash that bit of the code and write it again. It may not be broadly viable just yet, but it will be if the models keep getting cheaper and better. This is also implied in the idea of "vibe coding" - don't bother understanding the code or debugging it yourself; if it doesn't work the way you like, just say it and have the model fix it until it gets it right (or you run out of money).

Apocryphon 2y ago
Sounds very maintainable and rigorously constructed
- jryle70 2y ago
  How do you know it won't be maintainable? based on your current knowledge?
  - Apocryphon 2y ago
    Generating tons of throwaway code that is swapped out every time a bug occurs does not sound like engineering for the long-term.
    - TeMPOraL 2y ago
      Since when does any business care about "engineering for the long-term"? We, the software engineers, care. The business doesn't. In fact, the industry has systematically been trying to beat the care out of engineers - it's unprofessional to care about the work beyond the point it stops making money for the business. I'm not saying this is right or wrong - but this is how companies roll; if they can fix product issues by having AI throw chunks of code away and do it again, if that's reliably cheaper than having engineers do actual engineering, then that's what businesses will do.
      - Apocryphon 2y ago
        What happens when this leads to high-profile disasters? https://news.ycombinator.com/item?id=43449642 https://news.ycombinator.com/item?id=43449642
        TeMPOraL 2y ago
        Hopefully it won't. You don't put webshit in control of rockets or cars, and you shouldn't put vibecoded software in control of them either. Programming safety-critical systems is its own thing, and it should be resistant to LLM incursions at least as long as human sign-off is an important part of the job.
Jensson 2y ago
> LLMs don't just make it easy to accumulate code - they make it easy to throw code away and start again. To throw away code you have to understand it, so no its the opposite. Code you don't understand is the hardest to get rid of, so it stays the longest in your codebase. > if there's a bug and it can't be trivially solved, trash that bit of the code and write it again How do you know where the bug is if you don't understand the code? There is no known algorithm to take a bug description and return the place in the code the bug is, otherwise bug fixing would be trivial. Edit: Not to mention that in real production systems your bugs will corrupt the database, and if you haven't set up a logging system etc you will likely not realize for a while forcing you to do a rollback to a very old state losing so much data. You wont last long doing that.
- TeMPOraL 2y ago
  > To throw away code you have to understand it, so no its the opposite. Code you don't understand is the hardest to get rid of, so it stays the longest in your codebase. No, you don't. > How do you know where the bug is if you don't understand the code? There is no known algorithm to take a bug description and return the place in the code the bug is, otherwise bug fixing would be trivial. Yes, there is. At worst, the bug is somewhere in the entire project. But you probably have a more narrow idea where the bug is, or when it was introduced. "In module X", "In feature Y", "In the last N days/weeks". Not to mention, for most bugs, `git bisect` is enough to precisely narrow down the problematic change, and doing that doesn't actually require understanding anything about the code. It all boils down to costs. Even if it takes AI a whole day and 1000 attempts to do what would be a relatively simple fix for an experienced developer, if those 1000 attempts cost less than the developer's work-hours, the business will soon learn to prefer AI over people. If and when we get to that point, is mostly just a function of LLM performance and cost. If they get cheap enough, it'll make as much sense to have human developers fix bugs in code, as it makes sense for you to mend holes in your socks instead of buying them in bulk on-line. > Not to mention that in real production systems your bugs will corrupt the database, and if you haven't set up a logging system etc you will likely not realize for a while forcing you to do a rollback to a very old state losing so much data. This really depends on the kind of system you're doing, and the kind of data you're storing.
Panzer04 2y ago
Ever heard of chesterton's fence? If you don't understand what the code is doing, removing it sounds like a recipe for disaster.
- TeMPOraL 2y ago
  Have you ever heard of shirts? In the past, if you'd tear your shirt, you'd spend time mending it, or pay someone to do it for you. Today, you just throw it in the trash and buy a new one, as it's much cheaper and faster. Think of any other goods we don't repair anymore. Regardless of their internal complexity and beauty of engineering, and no matter how small the defect is, if it's cheaper to replace it wholesale than to repair it, people end up replacing it. There's no reason to believe the same won't happen to code.
  - ModernMech 2y ago
    That's only cheaper because we've made it someone else's problem. Someone is paying that cost -- fast fashion is a real issue that causes waste, pollution, and human rights violations. Now as we face tarriffs on foreign manufactured goods, things don't seem so cheap anymore. Mending socks looks a little better. You're right, there's no reason to believe the same won't happen to code, but there's also no reason to believe it won't similarly end in all kinds of problems that come back to bite us down the line when the goodtimes are over.
anon373839 2y ago
> if there's a bug and it can't be trivially solved, trash that bit of the code and write it again When I was in undergrad, I knew a few guys who approached every problem by pasting in snippets from Stack Overflow and tutorial sites until the code “worked”. Did not end well…
- TeMPOraL 2y ago
  That was then, and this is now. Look, I understand the sentiment. I too want the code to be done properly, well-engineered and thought through. But recall, this is not our job. We are Professionals, and by modern definition, a Professional does whatever is best for the business. And the business doesn't care about the product - it cares about the product's ability to earn them money. If generating shit code, then throwing it away and generating anew at any sign of bug (or spec change) gets cheap enough, this is what the business will want to do. This is what being Professional will mean. (You can imagine I don't hold "professionalism" in a very high regard.) See also: basic goods in meatspace. In developed economies, people generally don't repair clothes anymore, and increasingly rarely anyone bothers with repairing appliances. It's cheaper to just throw it away and buy a new one, than to try and repair it. Hell, even construction and remodeling these days involves a lot more of "affix it here permanently; if you need to move it, just smash it and install a new one" approach. Why wouldn't the same eventually happen with code?
  - deleted 2y ago
    [deleted]
  - anon373839 2y ago
    This is an interesting thought experiment, but I don’t think it’s realistic. Software that “works” doesn’t necessarily work. And software that doesn’t work can cost catastrophic financial losses. Worse yet, those losses are going to be felt more painfully when they are incurred recklessly. There is a reason that humans developed analytical problem solving as an alternative to trial and error: when you can do it, it’s more effective, and safer. That’s not to say that disposable code doesn’t have some interesting implications! One of them is that experimentation becomes a lot cheaper, so it’s faster to navigate the search space of possible solutions to a problem. But just taking a “solution” directly from an LLM without validating its correctness is fundamentally unserious and will be punished by reality sooner or later.

lukeschlather 2y ago

o1 has lately seemed more useful for debugging than for actually writing code. With writing code, it may give me some scaffold but I have a pretty particular idea of what I want and I have to rewrite most of it. With debugging, there have been multiple cases where I was stuck on something, I described the problem in detail and o1 gave me an insightful explanation of what I didn't understand. It's not magic, but mostly what makes it not magic is it can't respond to very detailed questions. But if I can get it the relevant information into a small space it can draw connections I can't.

cameldrv 2y ago

I think probably people will also use LLMs to debug. At this point, I think you can consider vibe-coding with an LLM to be pretty equivalent to using a fairly junior developer with access to stack overflow. It's going to make a lot of mistakes, it's going to make a lot of questionable decisions. Sometimes it will be able to fix its mistakes, sometimes it will spin its wheels and never fix it. It may make a big ball of mud. The entire project may fail. Two-three years from now, who knows. I'm pretty sure you still won't see 100% reliability in translating English->code, but you also don't see that even with senior developers.

corysama 2y ago
> I think probably people will also use LLMs to debug. I don’t have a link handy, but someone has already set up a service that’s “JS time travel debugger + LLM that knows how to use it” Pretty sure it was a Show HN recently.
- istjohn 2y ago
  https://news.ycombinator.com/item?id=43023698 https://news.ycombinator.com/item?id=43023698