4 ms·

For the past few days, I have been trying to fix a bug in a closed-source Mac app. I otherwise love the app, but this bug has been driving me crazy for years.

For the past few days, I have been trying to fix a bug in a closed-source Mac app. I otherwise love the app, but this bug has been driving me crazy for years. I was pretty sure I knew which Objective-C method was broadly responsible for the bug, but I didn't know what that method did, and the decompiled version was a nonsensical mess. I felt like I'd hit a wall. Then I thought to feed the decompiler babble to GPT-4 and ask for a clean version. The result wasn't perfect, but I was able to clean it up. I swizzled the result into the app, and I'm pretty sure the bug is gone. (I never found reproduction steps, but the problem would usually have occurred by now.) I never could have done this without GPT-4.

HarHarVeryFunny 3y ago
This sounds rather like the junior/bad developer who makes a bug disappear (at least for time being) by changing the order of functions in a source file or some such. Admittedly a complete rewrite of a piece of code, even without understanding what you are doing (e.g by using an LLM), is unlikely to have the same bugs as the original implementation (but may have different bugs), but hopefully no-one is doing this for code where bugs have any significant consequence (e.g. system downtime, cost to customers).
- Wowfunhappy 3y ago
  Just to be clear, I do understand the new version of the method. I don't entirely know how it fits into the larger system, but that's to be expected when I literally don't have the source code. When I cleaned it up, I took out some complexity which I believe was responsible for the bug, at the cost of some performance. According to GPT-4, the original version was checking file descriptors to decide when to do work. My version just does the work every 5ms.
  - ruszki 3y ago
    So the parent commenter tried to tell you, that they (and I too) heard this story from junior and bad programmers in the past 20 years all the time, and they didn’t use LLMs. It doesn’t matter whether you use generative AIs or not, it’s a bad way of thinking, and long term it’s not beneficial to anybody. The real problem is that you didn’t dig deeper when you figured out that that code change fixed the problem.
    - Wowfunhappy 3y ago
      But I actually think this is a reasonable way to fix a hard-to-pin-down bug in any context, at least temporarily. (In my case, I don't intend to go back because it's not my app and mostly for personal use, but that's beside the point.) There was a tradeoff between performance and complexity. The high-performance, high-complexity version was buggy, so I switched to a simpler option at the cost of some performance. This isn't where the LLM was significant. The LLM was able to make sense of unreadable decompiled code, similar to how the author had ChatGPT translate from compiled assembly code back to C. (Giving GPT-4 the actual assembly never occurred to me, in hindsight I should have tried that first.)
      - ruszki 3y ago
        My job is exactly to fix code which was not understood by its creators. And this “I have no idea why, but it works” (until it doesn’t) is the main cause of most of the problems which I work on. For example, at my current company the developer who introduced a “clever” navigation system didn’t know how HTML forms should be used, and why servers still allowed what they did. It worked. Now, 20+ years later that sole developer’s stupid decision, and lack of HTML best practices will cost my company a few million dollars (and by the way already cost probably a few million). A missing day of learning (and by the way a clear sign, that that developer should’ve never trusted with this task). Senior developers learn this, and I’ve never seen that better developers would be satisfied and would say “yeah, I fixed it”, when they don’t understand the what and how completely, even when it’s not strictly necessary. They burned themselves enough times.
        rictic 3y ago
        You've got me curious, how did the developer's misunderstanding of forms cost millions? Was it submitting duplicate orders? Blocking submission of valid orders causing lost business? I think it's an error to bring that up here though, where we're talking about someone patching a closed source app for their personal use. Is it worth the cost/benefit of decompiling and studying the app's code sufficiently long to be highly confident of the fix? Sloppiness and "good enough" has its place. So does full effort correctness.

HarHarVeryFunny 3y ago

This sounds rather like the junior/bad developer who makes a bug disappear (at least for time being) by changing the order of functions in a source file or some such. Admittedly a complete rewrite of a piece of code, even without understanding what you are doing (e.g by using an LLM), is unlikely to have the same bugs as the original implementation (but may have different bugs), but hopefully no-one is doing this for code where bugs have any significant consequence (e.g. system downtime, cost to customers).

Wowfunhappy 3y ago
Just to be clear, I do understand the new version of the method. I don't entirely know how it fits into the larger system, but that's to be expected when I literally don't have the source code. When I cleaned it up, I took out some complexity which I believe was responsible for the bug, at the cost of some performance. According to GPT-4, the original version was checking file descriptors to decide when to do work. My version just does the work every 5ms.
- ruszki 3y ago
  So the parent commenter tried to tell you, that they (and I too) heard this story from junior and bad programmers in the past 20 years all the time, and they didn’t use LLMs. It doesn’t matter whether you use generative AIs or not, it’s a bad way of thinking, and long term it’s not beneficial to anybody. The real problem is that you didn’t dig deeper when you figured out that that code change fixed the problem.
  - Wowfunhappy 3y ago
    But I actually think this is a reasonable way to fix a hard-to-pin-down bug in any context, at least temporarily. (In my case, I don't intend to go back because it's not my app and mostly for personal use, but that's beside the point.) There was a tradeoff between performance and complexity. The high-performance, high-complexity version was buggy, so I switched to a simpler option at the cost of some performance. This isn't where the LLM was significant. The LLM was able to make sense of unreadable decompiled code, similar to how the author had ChatGPT translate from compiled assembly code back to C. (Giving GPT-4 the actual assembly never occurred to me, in hindsight I should have tried that first.)
    - ruszki 3y ago
      My job is exactly to fix code which was not understood by its creators. And this “I have no idea why, but it works” (until it doesn’t) is the main cause of most of the problems which I work on. For example, at my current company the developer who introduced a “clever” navigation system didn’t know how HTML forms should be used, and why servers still allowed what they did. It worked. Now, 20+ years later that sole developer’s stupid decision, and lack of HTML best practices will cost my company a few million dollars (and by the way already cost probably a few million). A missing day of learning (and by the way a clear sign, that that developer should’ve never trusted with this task). Senior developers learn this, and I’ve never seen that better developers would be satisfied and would say “yeah, I fixed it”, when they don’t understand the what and how completely, even when it’s not strictly necessary. They burned themselves enough times.
      - rictic 3y ago
        You've got me curious, how did the developer's misunderstanding of forms cost millions? Was it submitting duplicate orders? Blocking submission of valid orders causing lost business? I think it's an error to bring that up here though, where we're talking about someone patching a closed source app for their personal use. Is it worth the cost/benefit of decompiling and studying the app's code sufficiently long to be highly confident of the fix? Sloppiness and "good enough" has its place. So does full effort correctness.