11 ms·
This is very similar in root cause and exploitation to Copy Fail. Which illustrates pretty well something that's lost when relying heavily on LLMs to do work f
by firer 5mo ago
This is very similar in root cause and exploitation to Copy Fail.
Which illustrates pretty well something that's lost when relying heavily on LLMs to do work for you: exploration.
I find that doing vulnerability research using AI really hinders my creativity. When your workflow consists of asking questions and getting answers immediately, you don't get to see what's nearby. It's like a genie - you get exactly what you asked for and nothing more.
The researcher who discovered Copy Fail relied heavily on AI after noticing something fishy. If he had to manually wade through lots of code by himself, he would have many more chances to spot these twin bugs.
At the same time, I'm pretty sure that by using slightly less directed prompting, a frontier LLM would found these bugs for him too.
It's a very unusual case of negative synergy, where working together hurt performance.
- eqvinox 5mo agoNo, unless I'm misreading it it's the *same* root cause: high 32 bits of Extended ESN in IPsec == authencesn module/cipher mode. The wrong thing got fixed for copy.fail, because people jumped to blame AF_ALG. [ed.: yes it's the same authencesn issue. https://github.com/V4bel/dirtyfrag/blob/892d9a31d391b7f0fccb333855f6289507186748/exp.c#L210 https://github.com/V4bel/dirtyfrag/blob/892d9a31d391b7f0fccb... it doesn't say authencesn in the code, only in a comment, but nonetheless, same issue.] [ed.2: the RxRPC issue is separate, this is about the ESP one]
- firer 5mo agoThere are two vulnerabilities here. The RxRPC one is definitely a different root cause (although caused by a very similar mistake). For the ESP one it's a bit harder to tell. I don't think the wrong thing was fixed, just that there was a very similar bug in almost the same spot. Could be wrong about that though.
- eqvinox 5mo ago(you probably wrote this while I was editing my post.) It's absolutely the same issue in authencesn/ESP. There's another one in RxRPC that is AIUI completely unrelated.
- pepa65 5mo agoBut if dirtyfrag is mitigated, copyfail is still active...
- formerly_proven 5mo agoThese are all page cache poisoning attacks (dirtyfrag, copyfail, dirtypipe). Maybe the page cache should have defense-in-depth measures for SUID binaries?
- firer 5mo agoSUID mitigations have nothing to do with the vulnerability itself - just the exploit. If there's a root cronjob that runs a world readable binary, you could modify it in the page cache and exploit it that way. Modifying the page cache is a really strong primitive with countless ways to exploit it.
- eqvinox 5mo agosplice() should maybe generally refuse to operate on things you can't write to.
- toast0 5mo agosplice is documented to return EBADF if "One or both file descriptors are not valid, or do not have proper read-write mode." So it seems surprising to me that you can call it when the out fd is not writable? But I didn't retain the information about the vulnerability, so I'm missing something. There was something about copy on write, IIRC?
- eqvinox 5mo ago"proper read-write mode" for the input fd is reading only. The exploit is writing to the splice() input fd. Also, NB, I said permission check, not mode check. The input fd to splice can and will be open for only reading quite often. Doesn't mean the kernel can't still do a write permission check. (Except I didn't say that here. Oops. Getting confused with my posts.)
- toast0 5mo agoOK, I may likely have too much sleep debt to understand, but given the bug is that splice can write to the input fd, you're suggesting maybe splice should only let you use an input fd if the process has access to write to it? But splice is a more or less a generalization of sendfile, and sendfile is often used for webserving where the serving process does not have ownership of the documents it is serving. It doesn't make sense to limit splice such that it can't do the task it was built for. Maybe splice should just not write to the input fd? :P
- tptacek 5mo agoI don't follow. LLMs spotted these bugs in the first place. You seem to be saying that these discoveries are indications that they're bad for vulnerability discovery.
- firer 5mo agoFrom what I understand, the copy fail bug was found by researcher who noticed something weird and then using AI to scan the codebase for instances where that becomes a problem. I bet that with a slightly looser prompt/harness, the LLM could have found these twin bugs too. Yet at the same time, I also think that if the human researcher had manually scanned the code, he'd have noticed these bugs too. FWIW I do think LLMs are great tools for finding vulnerabilities in general. Just that they were visibly not optimally applied in this case.
- aerodexis 5mo agoThey could also have found all these things at the same time - and are slow-rolling the disclosures.
- eqvinox 5mo agoI don't think the copy.fail people understood the issue they found, as is evident by the heavy focus on AF_ALG/aead_algif, which is essentially "innocent" as we're seeing here. I think LLMs are great for vulnerability discovery, but you need to not skimp on the legwork and understanding what even you just found there.
- papascrubs 5mo agoOr a follow up prompt: "find similar classes of bugs". Once the actual case has been layed out finding like bugs isn't too hard. I hear you on the creativity bit. Like any tool, AI can put blinders on. Using it to augment without it fully taking over your workflow is tough.
- dgellow 5mo agoNot just like any tool though. Interacting with agents can be incredibly boring and frustrating in a way that I personally do not experience with other technology
- varispeed 5mo ago> When your workflow consists of asking questions and getting answers immediately, you don't get to see what's nearby. That's why is very very important to just step out and use saved time to go for a walk, to a park, sit on a bench, listen do birds, close eyes and zoom out. The state we are in is actually brilliant.
- refulgentis 5mo agoIt’s very hard to see a root vuln similar to, but not the same as, another discovered by AI, as a lesson about AI not exploring. Is there a counterfactual where you would say it explored well enough, besides both vulnerabilities published as one?
- deleted 5mo ago[deleted]
- SubiculumCode 5mo agoEvidence or are you just riffing?
- timcobb 5mo ago> When your workflow consists of asking questions and getting answers immediately, you don't get to see what's nearby. Very much aligns with my experience. For me this is the most unsatisfying thing about AI-based workflows in general, they miss stuff humans would never miss. All the time I wonder what am I missing that's right nearby? It's remarkable how many times I have to ask Claude code to fully ingest something before it actually puts it into context. It always tries to laser through to target it's looking for, which is often not what you want it to look for, at least not all you want it to look for. Getting these models to open up their field of vision is tough.
- deleted 5mo ago[deleted]
- ulrikrasmussen 5mo agoDo you think this is inherent or an artifact of prompting? Curiosity and side quests leads to higher token usage and longer time to finish, so I could understand why current harnesses and system prompts would not encourage that sort of thing. But what if a coding agent was prompted to be more curious during development? Like a human developer, make mental notes of alternatives to try out and chase suspicious looking code which may seem unrelated to the task at hand. It could even spawn rabbit hole agents in parallel. Taking a step back, this probably highlights major hazard with the increased usage of LLMs for coding, which is that everyone's style of work is going to converge because most code will be written by the 2-3 most popular models using the same system prompts.
- lloeki 5mo agoI've seen something similar, solutions generated feel very pythonic or javaesque in languages that are neither Python nor Java (C, Rust, Ruby) I've had to explicitly direct the machine to read existing sibling code and follow the specific idioms and patterns in use.
- timcobb 5mo ago> Do you think this is inherent or an artifact of prompting? Not sure! I mean, look at this sibling comment for example: https://news.ycombinator.com/item?id=48062797 https://news.ycombinator.com/item?id=48062797. Not my experience, but apparently others have this experience. > But what if a coding agent was prompted to be more curious during development? I've tried using the language of curiosity. My qualitative take was that it did have a positive impact, but not much. And I can only tinker with system prompting so much, before I get drawn into LLM driving :) > which is that everyone's style of work is going to converge yeah I imagine even people's styles of thinking will converge as a result of this, more so than from reading other people's prose or programs. I think I saw something on HN to this effect within the last month, too.
- riedel 5mo agoJust on a side note. Negative synergy does not seem so uncommon with machine learning. We did some research maybe 10 yrs ago an human/ML based duplicate detection (for a municipal support ticket system) . Research showed that pure AI and pur human outperformed co-working. Human oversight often e.g. overcorrected machine work. I think it is a nice HCI problem to solve actually to amplify creativity and unique skills in such processes. Particularly if they can be to some degree repetitive and tiresome.
- harshreality 5mo agoI don't know... after they found a high profile bug like copyfail, I wouldn't attribute not looking for similar bugs to them being overly dependent on AI. It's easy to stop exploring, for a while at least, after you've struck on a major find. Maybe they would've returned to it in a few months. It certainly inspired others to explore similar areas and find these new bugs. Isn't that enough?
- totallyrandom__ 5mo agoWhere does it say that the Dirty Frag one was found by LLMs? The research site with all the information doesn't mention LLM or AI at all.
- zimbatm 5mo agoMaybe. This phenomenon of security holes being found closely to each other was also common before LLMs. People attention gets directed to a place and typically more issues are getting found.
- YmiYugy 5mo agoAI or not, it’s always been reasonable common that a bunch of related vulnerabilities get discovered after shortly after the original one.