10 ms·

The higher the productivity multiplier towards exploiting software, the more developers would find themselves severely outmatched: exploiting software is someon

The higher the productivity multiplier towards exploiting software, the more developers would find themselves severely outmatched: exploiting software is someone’s full-time job, whereas the engineers already have one—building it. To express this in numerical terms, let’s consider developer’s incentive to spend effort learning to find and actually finding vulnerabilities in their software (as oppposed to building it) as D, and attacker’s incentive to spend effort exploiting that software as A. I would say initially A = D × 5 is fair. On one hand, the developer knows their code better. However, their code is open, and most software engineers by definition prefer building (otherwise they would have been pentesters) so that’s where most of their time is going. This is not news, of course, and has been so since forever. The newer factor is attackers working for nation-states, being protected by them, and potentially having figurative guns to their heads or at least livelihoods depending on the amount of damage they can deal; the lack of equivalent pressure on the developer’s side leads me to adjust it to A = D × 10. ×10 is our initial power differential between the attacker and the developer. Now, let’s multiply that effort by a constant L, reflecting the productivity boost from LLMs. Let’s make it a 10 (I’m sure many would say LLMs make them more tham ×10 more productive in exploit-finding, but let’s be conservative). Additionally, let’s multiply that by a variable DS/AS that reflects developer’s/attacker’s skill at using LLMs in such particular ways that find the most serious vulnerabilities. As a random guess, let’s say AS = DS × 5, as the attacker would have been exclusively using LLMs for this purpose. With these numbers substituted in, X would be our new power differential: X = (A × L × AS) ÷ (D × L × DS) X = (D × 10 × 10 × DS × 5) ÷ (D × 10 × DS) X = 50. If my math is right, the power differential between the attacker and a developer jumps from 10 to 50 in favour of the attacker. If LLMs ×100 the productivity, the new differential would be 500. I didn’t account for the fact that many (especially smaller) developers may not even have the resources to run the equivalent compute power as a dedicated hacking team. Some ways to shift the balance back could be ditching the OSS model and going all-in on the so-called “trusted computing”. Both measures would increase the amount of effort (compute) the attacker may need to spend, but both happen to be highly unpopular as they put more and more power and control in the hand of the corporations that build our computers. In this way, the rise of LLMs certainly advances their interests.

pron 6mo ago
> exploiting software is someone’s full-time job, whereas the engineers already have one—building it. But the attackers needs to spread their attack over many products, while the engineers only need to defend one. > The newer factor is attackers working for nation-states, being protected by them, and potentially having figurative guns to their heads or at least livelihoods depending on the amount of damage they can deal; the lack of equivalent pressure on the developer’s side leads me to adjust it to A = D × 10. Except that's true even without LLMs. LLMs improve both sides' capabilities by the same factor (at least hypothetically). > Additionally, let’s multiply that by a variable DS/AS that reflects developer’s/attacker’s skill at using LLMs in such particular ways that find the most serious vulnerabilities. As a random guess, let’s say AS = DS × 5, as the attacker would have been exclusively using LLMs for this purpose. I'm not sure that's right, because once attackers develop some skill, that skill could spread to all defenders through tools with the skill built into them. So again, we can remove the "LLM factor" from both sides of the equation. If anything, security skills can spread more easily to defenders with LLM because without LLMs, the security skill of the attackers require more effort to develop.
- strogonoff 6mo ago
  > > exploiting software is someone’s full-time job, whereas the engineers already have one—building it. > But the attackers needs to spread their attack over many products, while the engineers only need to defend one. Are you assuming every piece of software has a dedicated defender team? Strikes me as unlikely. Realistically, you have people whose job or passion is to develop software, who often work not on one but on N projects at the same time (especially in OSS), and who definitely aren’t going to make finding vulnerabilities their full-time job because if they do then there’ll be no one to build the thing in the first place. > Except that's true even without LLMs. Of course. That’s why I put it before I started taking into account LLMs. LLMs multiply the pre-existing imbalance. > once attackers develop some skill, that skill could spread to all defenders through tools with the skill built into them Sure, that’s an interesting point. I’m sure the attackers try to conceal their methods; the way we tend to find out about it is when an exploit is exhausted, stops being worth $xxxxxxxx, and starts to be sold on mass markets, at which point arguably it’s a bit late. Furthermore, you still mention those mystical “defenders”, as if you would expect an average software project to have any dedicated defenders. (Edited my reply to the latest point, I didn’t read it correctly the first time.)

pron 6mo ago

> exploiting software is someone’s full-time job, whereas the engineers already have one—building it. But the attackers needs to spread their attack over many products, while the engineers only need to defend one. > The newer factor is attackers working for nation-states, being protected by them, and potentially having figurative guns to their heads or at least livelihoods depending on the amount of damage they can deal; the lack of equivalent pressure on the developer’s side leads me to adjust it to A = D × 10. Except that's true even without LLMs. LLMs improve both sides' capabilities by the same factor (at least hypothetically). > Additionally, let’s multiply that by a variable DS/AS that reflects developer’s/attacker’s skill at using LLMs in such particular ways that find the most serious vulnerabilities. As a random guess, let’s say AS = DS × 5, as the attacker would have been exclusively using LLMs for this purpose. I'm not sure that's right, because once attackers develop some skill, that skill could spread to all defenders through tools with the skill built into them. So again, we can remove the "LLM factor" from both sides of the equation. If anything, security skills can spread more easily to defenders with LLM because without LLMs, the security skill of the attackers require more effort to develop.

strogonoff 6mo ago
> > exploiting software is someone’s full-time job, whereas the engineers already have one—building it. > But the attackers needs to spread their attack over many products, while the engineers only need to defend one. Are you assuming every piece of software has a dedicated defender team? Strikes me as unlikely. Realistically, you have people whose job or passion is to develop software, who often work not on one but on N projects at the same time (especially in OSS), and who definitely aren’t going to make finding vulnerabilities their full-time job because if they do then there’ll be no one to build the thing in the first place. > Except that's true even without LLMs. Of course. That’s why I put it before I started taking into account LLMs. LLMs multiply the pre-existing imbalance. > once attackers develop some skill, that skill could spread to all defenders through tools with the skill built into them Sure, that’s an interesting point. I’m sure the attackers try to conceal their methods; the way we tend to find out about it is when an exploit is exhausted, stops being worth $xxxxxxxx, and starts to be sold on mass markets, at which point arguably it’s a bit late. Furthermore, you still mention those mystical “defenders”, as if you would expect an average software project to have any dedicated defenders. (Edited my reply to the latest point, I didn’t read it correctly the first time.)