4 ms·

What makes you think RCEs are being found & fixed at a rate that’s faster than they’re being introduced? I could see it going either way.

by e28eta 14d ago

user43928 14d ago
Why would the model not find the vulnerability during implementation or testing before release? If it requires a lot of compute and trying, this is something that could be provided for common software.
- xboxnolifes 14d ago
  Because it's far cheaper to to not spend the tokens finding the vulnerabilities, and software is now being created and released magnitudes faster than ever before. I could see the huge software companies maybe having fewer vulnerabilities, but I expect to see so much more in the smaller side of things.
- wood_spirit 14d ago
  Sad that this could well be that the path to OpenAI and Anthropic profitability of this arms race between defending LLM white hatting a company’s website and the black hat LLMs attacking it? So the whole thing is forcing the good guys to outspend on tokens to preemptively defend against the risk of the bad guys outspending them on tokens, rather than buying tokens to actually add features to the product etc. So are they creating a market for the solution by helping create the problem? A kind of rent-seeking AI security-industrial complex!!
  - agileAlligator 14d ago
    The only thing AI has changed is that it has dropped both: the cost of attack and the cost of defense. Nothing in the game has materially changed; the game has just sped up.
    - wood_spirit 14d ago
      Who gets rent has changed. It puts me in mind of cloudfare et al
      - agileAlligator 14d ago
        Not really. Actually, for the purposes of cybersecurity, local models are far superior. Both offense and defense.
    - emzo 14d ago
      The game has increased in scope.
      - agileAlligator 14d ago
        That is the direct effect of reduced cost. Jevon's paradox type effect: cost goes down demand goes up. You can AI-check so many more things that would be very time consuming earlier.
    - red-iron-pine 14d ago
      plenty more has changed. for example the barrier to being a skiddie is basically gone, and low-skill would be hackers can hit very hard. to develop a CVE into a KEV in 2017 might take 2-3 months with a skilled team of serious security engineers; now my intern can get into police radios without knowing anything about the underlaying technology, essentially on a whim. any random tier 1 IT drone who can define a VLAN can potentially hit as hard as that team of security engineers now
      - agileAlligator 14d ago
        Yeah that is what I said, cost has gone down.
  - adventured 14d ago
    The path to vast OpenAI profitability is trivial: advertising. Monetizing several hundred million users = $100+ billion ad network. 900 million active weekly users. Silicon Valley can do ad networks extraordinarily easily. Anybody doubting the ability of OpenAI to build an ad network around GPT will likely be embarassed in the near future. The path to substantial profitability for Anthropic is questionable. The Chinese LLMs threaten them by far the most of the three major US LLMs. The money for Anthropic is certainly not in $20-$200 subscriptions. And they don't have anywhere near the consumer potential that GPT does, in terms of unleashing an ad spigot. So how far will the API money scale while being undercut by China. OpenAI has to fight with Google for the ad business, they're specifically building Gemini to focus on consumer + search. Anthropic's business looks cute next to Google's search ad business (which is entirely at risk in this inflection). Meta looks like the biggest potential loser right now, ad dollars will be sucked out of the rotting Facebook network (not Instagram) and redirected to the rapidly expanding, hyper rich context LLM interaction. Advertising on Facebook will feel like running dumb banner ads on Excite in a few years, compared to what GPT will know about its users. People that think Chinese LLMs are a general threat, don't understand consumer destination services, which is what GPT's future is. China currently has nothing to threaten with in that realm. There is half a trillion dollars of advertising up for grabs.
    - disgruntledphd2 14d ago
      > Silicon Valley can do ad networks extraordinarily easily. This is just not true, building an effective advertising platform costs significant amounts of money, time and people. Remember that you need to hire a sales force for this, and sales scales linearly rather than sub-linearly like engineering. Additionally, you need to spend a lot of money dealing with fraud, fake and malicious ads. Furthermore, you need to figure out where to put the ads and how to rank them. Finally, advertising is a zero sum game (given that the internet has already killed lots of print & OOH advertising), so the only way to win is to better better/cheaper (preferably both) than Google/Meta/Amazon. Best of luck with that (although to be fair to OpenAI they did hire Fidji who knows a lot of this stuff from her time at Facebook). They don't have a Sheryl Sandberg type figure, and she was also really important in selling FB ads to large advertisers. Just looking at their leadership team I don't see anyone with a background in (successful) ads companies, so I'm pretty sceptical that they can build this out quickly enough to matter.
  - bigfatkitten 14d ago
    Assuming an equal level of impact per token spent, the scales have tipped in favour of the attacker. White hats are constrained by needing to pay for their own tokens, only using (expensive) vendors who meet governance and risk requirements etc. Black hats are free to take over accounts and steal services from wherever they can.
    - brookst 14d ago
      For white hats, how has the cost of a thorough security review changed since, say, five years ago?
      - bigfatkitten 14d ago
        The price has gone up if you’re getting AI to do it. In terms of finding low hanging fruit, reasonably good code scanning tools have been around for a while. The thing that’s changed for attackers is speed. The things that got you hacked yesterday are the same things getting you hacked today. Finding and weaponising things like memory corruption bugs required an enormous amount of relatively hard to find skill, and considerable time. An idiot can now throw tokens at the problem and have something they can reliably use within minutes or hours.
- techpression 14d ago
  Because people need to spend time and money on that, which they won’t. The implementation is cheap, the review and follow-up is not (speaking from a pure LLM only workflow). My ratio is around 1:2 currently, so twice as much time spent fixing vs building.
- philbo 14d ago
  > it requires a lot of compute This is one reason > and trying and this is the other.
- imhoguy 14d ago
  The surface of potential issues is growing with complexity of all connected parts of the system. That applies to not only software. To prevent issues you either spend proportional amount (dollars, tokens, hours) on testing or reduce complexity of the system.
nmlt 14d ago
Those companies that produce more RCEs than they close will sink and those that don’t won’t.
- bigfatkitten 14d ago
  If customers actually cared about this, Microsoft would’ve gone bust 20 years ago.
  - embedding-shape 14d ago
    People didn't store their entire life in the cloud and had every service connected with each other 20 years ago. People pay more attention today, and companies pay a lot more attention today. Of course, depends heavily on what country you live in.
    - bigfatkitten 14d ago
      Can you point to a single vendor where this has actually occurred? Customers say these things in response to a breach, but in practice they don’t lift a finger to actually change anything. Entra ID is full of design-level bugs that allow full tenant takeover, but nobody is abandoning M365 in droves. Windows has been a piece of shit for decades, and it’s still the default and dominant desktop platform. Equifax lost personal data for almost 150 million people in 2017, and they’re financially stronger than ever. Okta got thoroughly compromised two years in a row (2022 and 2023), and they’re still the global market leader in their space.
TacticalCoder 14d ago
> What makes you think RCEs are being found & fixed at a rate that’s faster than they’re being introduced? It could go either way but we're already at a point where successful exploits in some software (like Chrome) require an absurd amount of exploits to be chained to lead to an actual RCE. We've seen chains requiring more than ten exploits: not kidding. We'll learn to put more and more sandboxes / guards / checks / defensive techniques everywhere and then all that's going to be needed is for AI looking for security issues to find something ridiculous like 10% of all the actual issues to stop RCEs dead in their tracks. Also arguably the current SNAFU was expected: we fully knew hardly anyone was taking security seriously. Now: not so much. Many projects had tens and even hundreds of issues pointed to them. I think we'll see several things: projects beginning to take security seriously, defense in depth getting generalized and hence RCEs requiring ever more bugs/exploits to be chained to achieve anything, low-hanging fruits getting patched at an insane pace, new code being immediately checked, by LLMs, for not just low-hanging fruits but also more advanced security weaknesses, etc. We may also see things like the lost art of configuring firewalls making a comeback, the generalization of hardware security modules (where applicable), and even things offering physical guarantees, like time-bounded retrieval protocols, beginning to get used seriously. If I had to bet I'd say it shall go both ways: some projects are going to extremely sloppy and full of holes but others are going to get so secure nobody shall ever break them.
brookst 14d ago
I'd love to see data, but my intuition is that the average developer has access to dramatically better security reviews and far lower cost than ever. There's more software being written than ever so maybe raw numbers of RCE's could be up, but as a percentage, I'd really expect them to be down. Especially among any fairly common software, as all it takes is anyone working on it to get the idea to test.
- d4mi3n 14d ago
  What I’m seeing is more developers pushing more code of dubious quality without the ability to respond to feedback on said code. You can have the best security review in the world, but if the author of the code is not equipped to understand the feedback it ends up being a moot point. The challenge to me seems less technical and more cultural: how do we keep ourselves intellectually honest and engaged when we now spend the majority of our time orchestrating agents and outsourcing the design and thought processes?
  - fc417fc802 14d ago
    Hey claude, compare this security review to the current codebase and patch up anything that needs it.
- tyre 14d ago
  One issue for developers is that the most powerful models refuse to do comprehensive reviews. You can’t ask Fable 5.1 to find every exploit in your codebase, because that’s indistinguishable from what a bad actor would do.
- Timwi 14d ago
  > I'd love to see data, but my intuition is that the average developer has access to dramatically better security reviews and far lower cost than ever. Where? If I ask Claude to do a “security review” of my software, it gets blocked as a possible hacking attempt.

What makes you think RCEs are being found & fixed at a rate that’s faster than they’re being introduced? I could see it going either way.

by e28eta 14d ago

What makes you think RCEs are being found & fixed at a rate that’s faster than they’re being introduced? I could see it going either way.

user43928 14d ago

Why would the model not find the vulnerability during implementation or testing before release? If it requires a lot of compute and trying, this is something that could be provided for common software.

xboxnolifes 14d ago
Because it's far cheaper to to not spend the tokens finding the vulnerabilities, and software is now being created and released magnitudes faster than ever before. I could see the huge software companies maybe having fewer vulnerabilities, but I expect to see so much more in the smaller side of things.
wood_spirit 14d ago
Sad that this could well be that the path to OpenAI and Anthropic profitability of this arms race between defending LLM white hatting a company’s website and the black hat LLMs attacking it? So the whole thing is forcing the good guys to outspend on tokens to preemptively defend against the risk of the bad guys outspending them on tokens, rather than buying tokens to actually add features to the product etc. So are they creating a market for the solution by helping create the problem? A kind of rent-seeking AI security-industrial complex!!
- agileAlligator 14d ago
  The only thing AI has changed is that it has dropped both: the cost of attack and the cost of defense. Nothing in the game has materially changed; the game has just sped up.
  - wood_spirit 14d ago
    Who gets rent has changed. It puts me in mind of cloudfare et al
    - agileAlligator 14d ago
      Not really. Actually, for the purposes of cybersecurity, local models are far superior. Both offense and defense.
  - emzo 14d ago
    The game has increased in scope.
    - agileAlligator 14d ago
      That is the direct effect of reduced cost. Jevon's paradox type effect: cost goes down demand goes up. You can AI-check so many more things that would be very time consuming earlier.
  - red-iron-pine 14d ago
    plenty more has changed. for example the barrier to being a skiddie is basically gone, and low-skill would be hackers can hit very hard. to develop a CVE into a KEV in 2017 might take 2-3 months with a skilled team of serious security engineers; now my intern can get into police radios without knowing anything about the underlaying technology, essentially on a whim. any random tier 1 IT drone who can define a VLAN can potentially hit as hard as that team of security engineers now
    - agileAlligator 14d ago
      Yeah that is what I said, cost has gone down.
- adventured 14d ago
  The path to vast OpenAI profitability is trivial: advertising. Monetizing several hundred million users = $100+ billion ad network. 900 million active weekly users. Silicon Valley can do ad networks extraordinarily easily. Anybody doubting the ability of OpenAI to build an ad network around GPT will likely be embarassed in the near future. The path to substantial profitability for Anthropic is questionable. The Chinese LLMs threaten them by far the most of the three major US LLMs. The money for Anthropic is certainly not in $20-$200 subscriptions. And they don't have anywhere near the consumer potential that GPT does, in terms of unleashing an ad spigot. So how far will the API money scale while being undercut by China. OpenAI has to fight with Google for the ad business, they're specifically building Gemini to focus on consumer + search. Anthropic's business looks cute next to Google's search ad business (which is entirely at risk in this inflection). Meta looks like the biggest potential loser right now, ad dollars will be sucked out of the rotting Facebook network (not Instagram) and redirected to the rapidly expanding, hyper rich context LLM interaction. Advertising on Facebook will feel like running dumb banner ads on Excite in a few years, compared to what GPT will know about its users. People that think Chinese LLMs are a general threat, don't understand consumer destination services, which is what GPT's future is. China currently has nothing to threaten with in that realm. There is half a trillion dollars of advertising up for grabs.
  - disgruntledphd2 14d ago
    > Silicon Valley can do ad networks extraordinarily easily. This is just not true, building an effective advertising platform costs significant amounts of money, time and people. Remember that you need to hire a sales force for this, and sales scales linearly rather than sub-linearly like engineering. Additionally, you need to spend a lot of money dealing with fraud, fake and malicious ads. Furthermore, you need to figure out where to put the ads and how to rank them. Finally, advertising is a zero sum game (given that the internet has already killed lots of print & OOH advertising), so the only way to win is to better better/cheaper (preferably both) than Google/Meta/Amazon. Best of luck with that (although to be fair to OpenAI they did hire Fidji who knows a lot of this stuff from her time at Facebook). They don't have a Sheryl Sandberg type figure, and she was also really important in selling FB ads to large advertisers. Just looking at their leadership team I don't see anyone with a background in (successful) ads companies, so I'm pretty sceptical that they can build this out quickly enough to matter.
- bigfatkitten 14d ago
  Assuming an equal level of impact per token spent, the scales have tipped in favour of the attacker. White hats are constrained by needing to pay for their own tokens, only using (expensive) vendors who meet governance and risk requirements etc. Black hats are free to take over accounts and steal services from wherever they can.
  - brookst 14d ago
    For white hats, how has the cost of a thorough security review changed since, say, five years ago?
    - bigfatkitten 14d ago
      The price has gone up if you’re getting AI to do it. In terms of finding low hanging fruit, reasonably good code scanning tools have been around for a while. The thing that’s changed for attackers is speed. The things that got you hacked yesterday are the same things getting you hacked today. Finding and weaponising things like memory corruption bugs required an enormous amount of relatively hard to find skill, and considerable time. An idiot can now throw tokens at the problem and have something they can reliably use within minutes or hours.
techpression 14d ago
Because people need to spend time and money on that, which they won’t. The implementation is cheap, the review and follow-up is not (speaking from a pure LLM only workflow). My ratio is around 1:2 currently, so twice as much time spent fixing vs building.
philbo 14d ago
> it requires a lot of compute This is one reason > and trying and this is the other.
imhoguy 14d ago
The surface of potential issues is growing with complexity of all connected parts of the system. That applies to not only software. To prevent issues you either spend proportional amount (dollars, tokens, hours) on testing or reduce complexity of the system.

nmlt 14d ago

Those companies that produce more RCEs than they close will sink and those that don’t won’t.

bigfatkitten 14d ago
If customers actually cared about this, Microsoft would’ve gone bust 20 years ago.
- embedding-shape 14d ago
  People didn't store their entire life in the cloud and had every service connected with each other 20 years ago. People pay more attention today, and companies pay a lot more attention today. Of course, depends heavily on what country you live in.
  - bigfatkitten 14d ago
    Can you point to a single vendor where this has actually occurred? Customers say these things in response to a breach, but in practice they don’t lift a finger to actually change anything. Entra ID is full of design-level bugs that allow full tenant takeover, but nobody is abandoning M365 in droves. Windows has been a piece of shit for decades, and it’s still the default and dominant desktop platform. Equifax lost personal data for almost 150 million people in 2017, and they’re financially stronger than ever. Okta got thoroughly compromised two years in a row (2022 and 2023), and they’re still the global market leader in their space.

TacticalCoder 14d ago

> What makes you think RCEs are being found & fixed at a rate that’s faster than they’re being introduced? It could go either way but we're already at a point where successful exploits in some software (like Chrome) require an absurd amount of exploits to be chained to lead to an actual RCE. We've seen chains requiring more than ten exploits: not kidding. We'll learn to put more and more sandboxes / guards / checks / defensive techniques everywhere and then all that's going to be needed is for AI looking for security issues to find something ridiculous like 10% of all the actual issues to stop RCEs dead in their tracks. Also arguably the current SNAFU was expected: we fully knew hardly anyone was taking security seriously. Now: not so much. Many projects had tens and even hundreds of issues pointed to them. I think we'll see several things: projects beginning to take security seriously, defense in depth getting generalized and hence RCEs requiring ever more bugs/exploits to be chained to achieve anything, low-hanging fruits getting patched at an insane pace, new code being immediately checked, by LLMs, for not just low-hanging fruits but also more advanced security weaknesses, etc. We may also see things like the lost art of configuring firewalls making a comeback, the generalization of hardware security modules (where applicable), and even things offering physical guarantees, like time-bounded retrieval protocols, beginning to get used seriously. If I had to bet I'd say it shall go both ways: some projects are going to extremely sloppy and full of holes but others are going to get so secure nobody shall ever break them.

brookst 14d ago

I'd love to see data, but my intuition is that the average developer has access to dramatically better security reviews and far lower cost than ever. There's more software being written than ever so maybe raw numbers of RCE's could be up, but as a percentage, I'd really expect them to be down. Especially among any fairly common software, as all it takes is anyone working on it to get the idea to test.

d4mi3n 14d ago
What I’m seeing is more developers pushing more code of dubious quality without the ability to respond to feedback on said code. You can have the best security review in the world, but if the author of the code is not equipped to understand the feedback it ends up being a moot point. The challenge to me seems less technical and more cultural: how do we keep ourselves intellectually honest and engaged when we now spend the majority of our time orchestrating agents and outsourcing the design and thought processes?
- fc417fc802 14d ago
  Hey claude, compare this security review to the current codebase and patch up anything that needs it.
tyre 14d ago
One issue for developers is that the most powerful models refuse to do comprehensive reviews. You can’t ask Fable 5.1 to find every exploit in your codebase, because that’s indistinguishable from what a bad actor would do.
Timwi 14d ago
> I'd love to see data, but my intuition is that the average developer has access to dramatically better security reviews and far lower cost than ever. Where? If I ask Claude to do a “security review” of my software, it gets blocked as a possible hacking attempt.