4 ms·
This seems kind of crazy. If LLMs are so stunningly good at finding vulnerabilities in code, then shouldn't the solution be to run an LLM against your code afte
by ButlerianJihad 7mo ago
This seems kind of crazy. If LLMs are so stunningly good at finding vulnerabilities in code, then shouldn't the solution be to run an LLM against your code after you commit, and before you release it? Then you basically have pentesting harnesses all to yourself before going public. If an LLM can't find any flaws, then you are good to release that code.
A few years ago, I invoked Linus's Law in a classroom, and I was roundly debunked. Isn't it a shame that it's basically been fulfilled now with LLMs?
https://en.wikipedia.org/wiki/Linus%27s_law https://en.wikipedia.org/wiki/Linus%27s_law
- fwip 7mo agoIt's entirely possible to address all the LLM-found issues and get an "all green" response, and have an attacker still find issues that your LLM did not. Either they used a different model, a different prompt, or spent more money than you did. It's not a symmetric game, either. On defense, you have to get lucky every time - the attacker only has to get lucky once.
- earthnail 7mo ago> It's not a symmetric game, either. On defense, you have to get lucky every time - the attacker only has to get lucky once. This! I love OSS but this argument seems to get overlooked in most of the comments here.
- deleted 7mo ago[deleted]
- dgellow 7mo agoI mean, you should definitely have _some_ level of audit by LLMs before you ship, as part of the general PR process. But you might need thousands of sessions to uncover some vulnerabilities, and you don’t want to stop shipping changes because the security checks are taking hours to run
- samename 7mo agoThat’s a non-trivial cost for commonly severely underfunded open source projects
- yawndex 7mo agoCal.com is not a severely underfunded project, it raised around $32M of VC money.
- evanelias 7mo agoIt's not a "project" though; the business Cal.com Inc raised that VC money. Their open source repo did not raise the money. Did they ever promise to keep their codebase FOSS forever, in a way that differs from what they're already doing over at cal.diy? If not, I don't see why it would be reasonable to expect them to spend a huge amount of money re-scanning on every single commit/deploy in order to keep their non-"DIY" product open source.
- vlapec 7mo agoLLMs really are stunningly good at finding vulnerabilities in code, which is why, with closed-source code, you can and probably will use them to make your code as secure as possible. But you won't keep the doors open for others to use them against it. So it is, unfortunately, understandable in a way...
- paprikanotfound 7mo agoI'm not a security expert but can't close source applications be vulnerable and exploited too? I feel like using close source as a defense is just giving you a false sense of security.
- pixel_popping 7mo agoDelaying attacks is a form of valid security.
- layer8 7mo agoFinding a vulnerability in a black box is drastically different from finding one in a white box. This isn’t about whether there is a vulnerability or not, but about the likelihood of it being found.
- ygjb 7mo agoNo it isn't. There is a tooling gap, and there is a skill gap, but both of those are being rapidly closed by both open and closed source projects. LLMs, and tools built to use them, are violating a lot of assumptions these days.
- thombles 7mo agoIt's a meaningful difference for SaaS. Most likely an attacker doesn't have access to your running binary let alone source code, and if they probe it like a pentester would it will be noisy and blocked/flagged by your WAF.
- 7mo ago
- maxloh 7mo ago[dead]
- layer8 7mo agoAttackers only need LLMs to be good at randomly finding one vulnerability, whereas service providers need them to be good at finding all such vulnerabilities.
- johnfn 7mo agoAfter a release, attackers have effectively infinite time to throw an LLM against every line of your code - an LLM that only gets smarter and cheaper to run as time passes. In order to feel secure you’d need to do all the work you’d imagine an attacker would ever do, for every single release you ship.
- stavros 7mo agoThis assumes that the relationship between "LLM tokens spent" and "vulnerabilities found" doesn't plateau, though.
- mixdup 7mo agoThe first few times it's going to be expensive, but once everyone level sets with intense scans of their codebases, "every single release" is actually not that big a deal, since you are not likely to be completely rebuilding your codebase every release
- techpression 7mo agoYou still have to account for the non-deterministic behavior of an LLM, when do you know you have exhausted its possible outcomes for any given piece of code?
- johnfn 7mo agoI'm not sure. An innocuous one line change like "bump version" possibly adds a million new lines of code.
- utopiah 7mo ago> attackers have effectively infinite time No, attackers are also rational economical actors. They don't randomly attack any software just for the aesthetics beauty of the process. They attack for bounty, for fame, for national interest, etc. No matter the reason it's not random and thus they DO have a budget, both in time and money. They attack THIS project versus another project because it's interesting to them. If it's not, they might move to another project but they certainly won't spend infinite time precisely because they don't have infinite resources. IMHO it's much more interesting to consider the realistic arm race then theoretical scenarii that never take place.
- sandeepkd 7mo agoEvery change would introduce the possibility of a vulnerability being added to the system and one would need to run the LLM scan across the entire code base. It gets very costly in a environment where you are doing regular commits. Companies like Github already provide scanning tools for static analysis and the cost is already high for them.
- deleted 7mo ago[deleted]
- pianopatrick 7mo agoMight lead to a move away from continuous delivery back towards batched releases.
- r2vcap 7mo agoAs LLMs improve and adoption grows, maintaining a FOSS project is becoming more complex and more expensive in terms of time and manpower. That part is easy to understand. It is also become a trend that LLM-assisted users are generating more low-quality issues, dubious security reports, and noisy PRs, to the point where keeping the whole stack open source no longer feels worth it. Even if the real reason is monetization rather than security, I can still understand the decision. I suspect we will see more of this from commercial products built around a FOSS core. The other failure mode is that maintainers stop treating security disclosures as something special and just handle them like ordinary bugs, as with libxml2. In that sense, Chromium moving toward a Rust-based XML library is also an interesting development.
- d3Xt3r 7mo agoJust use AI to fight AI, that's the only sensible way we can keep up. So if you're low-quality PRs, reports etc, have LLMs filter them out. Like how once upon a time we used to drown in email spam but it's now mostly a non-issue thanks to intelligent spam filters, the same needs to happen for opensource projects. Use AI to fight AI.
- wartywhoa23 7mo agoIn other words, have more money to pay than your enemy. This game will end horribly.
- pcblues 7mo agoWrite simple code. Do what you said, which is a very good idea. Test LLM security against the compiler too.
- benjaminoakes 7mo agoI suspect that AI is a convenient excuse to go closed source. They have probably wanted to do that for years after leaning more commercial.