3 ms·

Rather than patiently referencing syscall tables by hand which is error prone and hard to review as a human, the LLM wrote tests to actually compare each of the

by lrvick 17d ago

Rather than patiently referencing syscall tables by hand which is error prone and hard to review as a human, the LLM wrote tests to actually compare each of these to the kernel syscall tables to make my life a lot easier and avoid regressions or mistakes. Complex tests to write, that saves me a lot of time as a security reviewer. Can change any value and watch tests fail. https://codeberg.org/lrvick/nit/src/branch/main/tests/kernel_source.rs https://codeberg.org/lrvick/nit/src/branch/main/tests/kernel... That is the point. Getting to results that satisfy security and functionality requirements in much less time. Also, I was able to generate a bare minimal and hardened kernel config by using the LLM to write a tool to trace all kernel calls. Something that I would have normally been forced to punt on that LLMs made easy to justify. LLMs are just as Linus Torvalds likes to say "Auto complete on steroids".

fwip 17d ago
The parsing code isn't even correct, though. I'm sure it works for the current input source code, but it's a clear sign of an LLM developing by iteration, rather than actually understanding the problem and coming up with a correct solution. How often does "this table of constants was transcribed incorrectly" even come up as a bug class? Is it worth writing a buggy C parser for a failure mode that doesn't happen? Do you expect that Linux is going to change their constant definitions out from under you?
- lrvick 16d ago
  Correct depends on what problem one is trying to solve though the parsing can certainly be cleaned up before release. It is a working WIP. > Do you expect that Linux is going to change their constant definitions out from under you Unlikely but I have been surprised before. Hardcoding to pretty much any modern kernel source version forever for this narrow use case is probably fine. The goal is about helping me and a reviewer quickly form confidence none of the magic numbers are mistaken or malicious in an automated way against the source of truth.

by lrvick 17d ago

fwip 17d ago

The parsing code isn't even correct, though. I'm sure it works for the current input source code, but it's a clear sign of an LLM developing by iteration, rather than actually understanding the problem and coming up with a correct solution. How often does "this table of constants was transcribed incorrectly" even come up as a bug class? Is it worth writing a buggy C parser for a failure mode that doesn't happen? Do you expect that Linux is going to change their constant definitions out from under you?

lrvick 16d ago
Correct depends on what problem one is trying to solve though the parsing can certainly be cleaned up before release. It is a working WIP. > Do you expect that Linux is going to change their constant definitions out from under you Unlikely but I have been surprised before. Hardcoding to pretty much any modern kernel source version forever for this narrow use case is probably fine. The goal is about helping me and a reviewer quickly form confidence none of the magic numbers are mistaken or malicious in an automated way against the source of truth.