3 ms·

IDK seems like a bug that could've taken a human a few minutes at best to find. I found a bug in SystemD that would crash the daemon because a bad SystemD unit

IDK seems like a bug that could've taken a human a few minutes at best to find. I found a bug in SystemD that would crash the daemon because a bad SystemD unit file configuration. That took me like 5 minutes to actually track down in the actual source code. I understand the utility of this though, I just don't see this particular bug and something that would be particularly difficult o find pre LLM era.

thephyber 1mo ago
This is the wrong mentality. The fuzzer found the bug before any humans did, so there is a mismatch of developers who could find this bug and those who did (without an LLM-coded fuzzer). The value of the fuzzer continues long after it found this one bug. It's worth nothing that in the bug discussion thread, the bug fix author pointed out that it's not easy to set up the config then call the functions in the right order. Your comment assumes that the reader has enough context to read the code and build the finite state automata in their mind. The bug fix reporter's comments suggest that you are assuming things which you shouldn't assume.
- written-beyond 1mo ago
  But I am being very specific to this use case, where a division by zero bug was found. Why couldn't you have just grepped through the codebase, found all possible divisions and ensured that they had a check on it to never be less than or equal to 0? The bug I located in SystemD was literally a null ptr exception. All they had to do was perform a null check on a cstring but they hadn't. I don't see the utility of reporting an LLM made fuzzer finding bugs that could be found by a lint rule or static analysis. I will appreciate a post about an LLM fuzzing software to find unique corner cases, which I predict will happen soon, in ACL controlled systems caused by policy shadowing.
pfdietz 1mo ago
Bugs are always easier to find in retrospect.
- written-beyond 1mo ago
  This is literally the lowest rung of a bug, it's equivalent to finding a nullptr exception. You can literally avoid them with 1 if statement.

thephyber 1mo ago

This is the wrong mentality. The fuzzer found the bug before any humans did, so there is a mismatch of developers who could find this bug and those who did (without an LLM-coded fuzzer). The value of the fuzzer continues long after it found this one bug. It's worth nothing that in the bug discussion thread, the bug fix author pointed out that it's not easy to set up the config then call the functions in the right order. Your comment assumes that the reader has enough context to read the code and build the finite state automata in their mind. The bug fix reporter's comments suggest that you are assuming things which you shouldn't assume.

written-beyond 1mo ago
But I am being very specific to this use case, where a division by zero bug was found. Why couldn't you have just grepped through the codebase, found all possible divisions and ensured that they had a check on it to never be less than or equal to 0? The bug I located in SystemD was literally a null ptr exception. All they had to do was perform a null check on a cstring but they hadn't. I don't see the utility of reporting an LLM made fuzzer finding bugs that could be found by a lint rule or static analysis. I will appreciate a post about an LLM fuzzing software to find unique corner cases, which I predict will happen soon, in ACL controlled systems caused by policy shadowing.

pfdietz 1mo ago

Bugs are always easier to find in retrospect.

written-beyond 1mo ago
This is literally the lowest rung of a bug, it's equivalent to finding a nullptr exception. You can literally avoid them with 1 if statement.