3 ms·

Why is it unreadable? I actually find LLM bug reports/breakdowns to be far more detailed and concise that classical human written ones. If you read the linked r

by rfgplk 3mo ago

Why is it unreadable? I actually find LLM bug reports/breakdowns to be far more detailed and concise that classical human written ones. If you read the linked repo it clearly goes it depth where the bug was found, how to reproduce it (and in depth). Most disclosures that are human written don't do this at all, they barely even tell you _how_ to reproduce the bug. Just look at the "3.3 The self-store tear", the LLM clearly describes exactly what went wrong, so you can verify it by hand.

walk12111 3mo ago
[dead]
skydhash 3mo ago
Because any writing needs a core intent they need to convey, which you can summarize down to according to the audience and why it should be important to them. Kinda like the same idea of elevator pitches, “explain like I’m five” and tactical reports when there’s a time constraints. You got none of that here. It’s just realms of text.
- rfgplk 3mo ago
  Maybe the person writing the report isn't an expert in this domain or doesn't have the time to commit to it? From my point of view as long as the information is accurate and reproducible, it's valuable.
  - mi_lk 3mo ago
    Then where’s the lie in “soulless AI slop”?
  - applfanboysbgon 3mo ago
    > From my point of view as long as the information is accurate That's the trillion-dollar catch, isn't it. LLMs love to write 30 paragraphs about some plausibly-correct-sounding explanation that is just as likely to be completely fucking wrong as it is accurate. The bug might be real, but that doesn't mean this analysis is accurate, and trying to figure out where the LLM went off the rails can be a nightmare. If you can actually understand the bug, it doesn't take 30 paragraphs to explain it. I would throw this bug report into my junk bin if I were on the receiving end of it, and I say that as someone who will spend days troubleshooting any issue a user will help me diagnose even if it only happens on their machine.
  - orphea 3mo ago
    > Maybe the person writing the report isn't an expert in this domain or doesn't have the time to commit to it? Then maybe that person should not do it? At least until they find the time?
    - fr2029 3mo ago
      [dead]
    - NetMageSCW 3mo ago
      So just spend a few years learning to be a kernel developer so they can report a kernel bug that is crashing their user program? That is so open source it’s not even funny.
      - fc417fc802 3mo ago
        No, just omit the rambling in favor of a "sorry I lack the expertise to really dig into this, here's a reproducer and I'm here if you have any questions I'm able to help with". Your bizarre overly literal take is so peak HN that it ... okay I admit it's still funny to me.
  - epixu 3mo ago
    I'd rather read walls of AI slop than soulful meatbag bickering. Just because you might have a soul and intent doesn't mean you do anything useful with it. All I see is the intentful invention of more reasons to fight over arbitrary crap.
    - lp196 3mo ago
      [dead]
  - neerajsi 3mo ago
    The other part is that almost no one would have the energy to spend hours instrumenting and rebuilding musl, reading the kernel mm code, and thinking hard about how they might interact. Claude is probably not right about the root cause here and is probably bsing, I agree. But it's collected enough raw data to point some expert humans at the right interaction. I'd take this bs bug report and start asking Claude some questions that would guide it to a more plausible actual answer, but that requires a little more experience in kernel development.
    - skydhash 3mo ago
      > The other part is that almost no one would have the energy to spend hours instrumenting and rebuilding musl, reading the kernel mm code, and thinking hard about how they might interact. Maybe because it isn’t needed. That’s the reason we have experts and professionals, because it’s more economical to use them than for everyone to start from scratch. It’s easier to go to a mechanic shops to fix my engine block than to try to do it myself. I’ve heard that a lot of shops charge more if you said an amateur try to fix things first. It’s a lot of stuff, but that’s not how you experiment for an hypothesis.
      - neerajsi 3mo ago
        I've been a systems dev for a long time and have debugged many such tricky issues before. Even on teams of experts, this type of issue would take someone really capable weeks to track down, especially because the repro environment is hard to obtain. And the opportunity cost of doing this type of work for a rare bug is high, since you may never get to an answer and have burned a lot of time. So in practice I solved most of this type of issue early in my career before I had a family and before I had higher level work to do. The ai doesn't have the same incentives as a human. It can also be applied to the repro where it exists. Your mechanic analogy is a bit inapt. Sure they're experts and can replace your engine much better than you can. But tracking down a non obvious system level bug like this one is much less repeatable and much harder because the problem has already been filtered through a lot of reviews and testing. As a system becomes mature, the bugs become harder and harder individually to track down.
jerf 3mo ago
I'm reasonably confident that the author has a reproduction case on their hands. That's easy to directly verify. I'm way less confident in that long and rambling analysis. As fwlr observes, it's not clear that this has been reproduced off the original machine: https://news.ycombinator.com/item?id=49134357 https://news.ycombinator.com/item?id=49134357 If it is the case that the original machine has an intermittent hardware fault, that analysis is exactly what I'd expect of an current AI. Voluminous and confidently wrong. Also interesting that the AI itself doesn't note this... having been biased by the conversation to consider kernel bugs, in the "cross-machine data" [1] it blames the kernel and doesn't appear to consider alternate explanations. It doesn't get mentioned much around here but one major issue with AIs today is that they fall into tunnel vision very easily and often need some help getting out of it. There's a structural reason for tunnel vision built into their context limits, and there's also situations like this, where if you bias them in a certain direction they can get monomaniacally stuck on it. You can also see it where they do things like try to do something on your system like view a certain file, then if for some reason they fail due to permission issues, if you don't stop them they'll go absolutely insane trying different ways to access the file, rather than just stopping and saying "Hey, I need help". Even if you prompt them directly to do that, it only helps, it doesn't fix the issue. I suspect the RHLF training they get to be better coding agents reinforces this behavior because in the coding quality tests they're rewarded for one-shotting all the various benchmarks, where that "bull in a china shop" approach works better than giving up. But I'd actually like them to give up a bit more often. I've had the same problem with some particularly go-getting junior devs, too... I appreciate the ambition and I look forward to harnessing it in other situations but I'd rather you didn't spend five hours to create a terrible work around to something I could have gotten you in two minutes. For the junior dev, it's OK, they take the feedback and adjust... the AIs never adjust. [1]: https://github.com/dfoxfranke/ripgrep-3494-analysis#6-cross-machine-data https://github.com/dfoxfranke/ripgrep-3494-analysis#6-cross-...
- skydhash 3mo ago
  That reminds me of one ticket I had to handle. It was generated by an agent from a customer report and the different hypotheses were wrong. Also a huge wall of text. The issue was alongside the API boundary (timezone shenanigans) but the agent invented a lot of reasons with reasonable looking arguments because of the tunnel vision that there should be something wrong inside the project.

by rfgplk 3mo ago

walk12111 3mo ago

[dead]

skydhash 3mo ago

Because any writing needs a core intent they need to convey, which you can summarize down to according to the audience and why it should be important to them. Kinda like the same idea of elevator pitches, “explain like I’m five” and tactical reports when there’s a time constraints. You got none of that here. It’s just realms of text.

rfgplk 3mo ago
Maybe the person writing the report isn't an expert in this domain or doesn't have the time to commit to it? From my point of view as long as the information is accurate and reproducible, it's valuable.
- mi_lk 3mo ago
  Then where’s the lie in “soulless AI slop”?
- applfanboysbgon 3mo ago
  > From my point of view as long as the information is accurate That's the trillion-dollar catch, isn't it. LLMs love to write 30 paragraphs about some plausibly-correct-sounding explanation that is just as likely to be completely fucking wrong as it is accurate. The bug might be real, but that doesn't mean this analysis is accurate, and trying to figure out where the LLM went off the rails can be a nightmare. If you can actually understand the bug, it doesn't take 30 paragraphs to explain it. I would throw this bug report into my junk bin if I were on the receiving end of it, and I say that as someone who will spend days troubleshooting any issue a user will help me diagnose even if it only happens on their machine.
- orphea 3mo ago
  > Maybe the person writing the report isn't an expert in this domain or doesn't have the time to commit to it? Then maybe that person should not do it? At least until they find the time?
  - fr2029 3mo ago
    [dead]
  - NetMageSCW 3mo ago
    So just spend a few years learning to be a kernel developer so they can report a kernel bug that is crashing their user program? That is so open source it’s not even funny.
    - fc417fc802 3mo ago
      No, just omit the rambling in favor of a "sorry I lack the expertise to really dig into this, here's a reproducer and I'm here if you have any questions I'm able to help with". Your bizarre overly literal take is so peak HN that it ... okay I admit it's still funny to me.
- epixu 3mo ago
  I'd rather read walls of AI slop than soulful meatbag bickering. Just because you might have a soul and intent doesn't mean you do anything useful with it. All I see is the intentful invention of more reasons to fight over arbitrary crap.
  - lp196 3mo ago
    [dead]
- neerajsi 3mo ago
  The other part is that almost no one would have the energy to spend hours instrumenting and rebuilding musl, reading the kernel mm code, and thinking hard about how they might interact. Claude is probably not right about the root cause here and is probably bsing, I agree. But it's collected enough raw data to point some expert humans at the right interaction. I'd take this bs bug report and start asking Claude some questions that would guide it to a more plausible actual answer, but that requires a little more experience in kernel development.
  - skydhash 3mo ago
    > The other part is that almost no one would have the energy to spend hours instrumenting and rebuilding musl, reading the kernel mm code, and thinking hard about how they might interact. Maybe because it isn’t needed. That’s the reason we have experts and professionals, because it’s more economical to use them than for everyone to start from scratch. It’s easier to go to a mechanic shops to fix my engine block than to try to do it myself. I’ve heard that a lot of shops charge more if you said an amateur try to fix things first. It’s a lot of stuff, but that’s not how you experiment for an hypothesis.
    - neerajsi 3mo ago
      I've been a systems dev for a long time and have debugged many such tricky issues before. Even on teams of experts, this type of issue would take someone really capable weeks to track down, especially because the repro environment is hard to obtain. And the opportunity cost of doing this type of work for a rare bug is high, since you may never get to an answer and have burned a lot of time. So in practice I solved most of this type of issue early in my career before I had a family and before I had higher level work to do. The ai doesn't have the same incentives as a human. It can also be applied to the repro where it exists. Your mechanic analogy is a bit inapt. Sure they're experts and can replace your engine much better than you can. But tracking down a non obvious system level bug like this one is much less repeatable and much harder because the problem has already been filtered through a lot of reviews and testing. As a system becomes mature, the bugs become harder and harder individually to track down.

jerf 3mo ago

I'm reasonably confident that the author has a reproduction case on their hands. That's easy to directly verify. I'm way less confident in that long and rambling analysis. As fwlr observes, it's not clear that this has been reproduced off the original machine: https://news.ycombinator.com/item?id=49134357 https://news.ycombinator.com/item?id=49134357 If it is the case that the original machine has an intermittent hardware fault, that analysis is exactly what I'd expect of an current AI. Voluminous and confidently wrong. Also interesting that the AI itself doesn't note this... having been biased by the conversation to consider kernel bugs, in the "cross-machine data" [1] it blames the kernel and doesn't appear to consider alternate explanations. It doesn't get mentioned much around here but one major issue with AIs today is that they fall into tunnel vision very easily and often need some help getting out of it. There's a structural reason for tunnel vision built into their context limits, and there's also situations like this, where if you bias them in a certain direction they can get monomaniacally stuck on it. You can also see it where they do things like try to do something on your system like view a certain file, then if for some reason they fail due to permission issues, if you don't stop them they'll go absolutely insane trying different ways to access the file, rather than just stopping and saying "Hey, I need help". Even if you prompt them directly to do that, it only helps, it doesn't fix the issue. I suspect the RHLF training they get to be better coding agents reinforces this behavior because in the coding quality tests they're rewarded for one-shotting all the various benchmarks, where that "bull in a china shop" approach works better than giving up. But I'd actually like them to give up a bit more often. I've had the same problem with some particularly go-getting junior devs, too... I appreciate the ambition and I look forward to harnessing it in other situations but I'd rather you didn't spend five hours to create a terrible work around to something I could have gotten you in two minutes. For the junior dev, it's OK, they take the feedback and adjust... the AIs never adjust. [1]: https://github.com/dfoxfranke/ripgrep-3494-analysis#6-cross-machine-data https://github.com/dfoxfranke/ripgrep-3494-analysis#6-cross-...

skydhash 3mo ago
That reminds me of one ticket I had to handle. It was generated by an agent from a customer report and the different hypotheses were wrong. Also a huge wall of text. The issue was alongside the API boundary (timezone shenanigans) but the agent invented a lot of reasons with reasonable looking arguments because of the tunnel vision that there should be something wrong inside the project.