3 ms·

This article is AI generated, but I kinda reject the premise that it's "not all it cracked up to be" just because the agents had reasons to act the way they did

This article is AI generated, but I kinda reject the premise that it's "not all it cracked up to be" just because the agents had reasons to act the way they did & were a result of human error. The hack still happened, it should still be a wake up call, we are going to start seeing this more frequently, and learning to defend against it is going to become more important over time. None of that is challenged by any particular reason for it happening, it still happened and it's still going to happen again. Threat models are going to have to start including that IPv4 (or whatever) scanners aren't necessarily going to only be spray and pray anymore, they could have relentless automated models at the other end that will literally dig into the particulars of your infrastructure looking for novel vulnerabilities to exploit. Maybe people will finally start to understand why security by obscurity has never been very reliable.

minimaxir 14d ago
Calling an article from the Wall Street Journal AI generated is a stretch.
- deleted 14d ago
  [deleted]
- LoganDark 14d ago
  I don't care who published it, this is clearly AI-written. > But the Hugging Face episode is different. It left logs, reports, design decisions and identifiable points at which human beings could have intervened. And that record suggests a less thrilling but more useful lesson. > People built the test, removed restraints, defined the objective, left a route open and decided not to stop what was happening. Calling the result "rogue AI" does more than sensationalize it. It allows those human decisions to disappear quietly from the story.
  - nkurz 13d ago
    I think your confidence is likely misplaced, but I vouched for your comment to revive it from the dead because you expressed your opinion clearly and with examples. To the flaggers: flag comments that you think violate the rules, but don't flag something just because you strongly disagree with the claim it makes.
    - LoganDark 12d ago
      It was [flagged] very quickly, too. I checked my screen capture just now, and it happened between 4 and 14 seconds after posting. Either I tripped something automated, or someone with incredible flagging powers was sitting there waiting to immediately suppress my reply. Was a little disheartening to see my supporting evidence hidden for so long.
ozozozd 14d ago
You are not referring to a port scanner, right?
- LoganDark 14d ago
  I guess. The IPv4 address space is small enough that it's been feasible to scan the entire thing exhaustively for a while now. Before LLMs, a public IPv4 would mostly get automated scripts that try the same things on every address, sometimes depending on what port scans find, yes. But we're going to start seeing more adversaries that have LLMs investigate each individual address to find novel or unique vulnerabilities. You can spray individualized reverse engineering without having to dedicate a real reverse engineer's time to each target. Avoiding the attention of attackers won't be enough to get away with mere security by obscurity.
  - tancop 13d ago
    You can use the same RE tools and run your own agents as a defender. Everything they find helps you harden your setup, to the point there might be no exploitable vulns at all after a couple days of intense red teaming. It's not completely bulletproof (unless you do formal verification, another thing LLMs are good at) but the network boundary stops a lot of attack vectors. You can't really do side channel or timing attacks over IP, so all that's left are easier to patch logic bugs.
    - LoganDark 13d ago
      > You can't really do side channel or timing attacks over IP Ehh, this has been disproven dozens of times but that's besides the point, I think. I absolutely agree that defenders should be using every tool at their disposal to harden their infrastructure. A lot of defenders simply don't do that, and that's always been a shame. I truly hope that this increasing threat leads to better defensive effort. People need to realize they can't just get away with it the same as before.

minimaxir 14d ago

Calling an article from the Wall Street Journal AI generated is a stretch.

deleted 14d ago
[deleted]
LoganDark 14d ago
I don't care who published it, this is clearly AI-written. > But the Hugging Face episode is different. It left logs, reports, design decisions and identifiable points at which human beings could have intervened. And that record suggests a less thrilling but more useful lesson. > People built the test, removed restraints, defined the objective, left a route open and decided not to stop what was happening. Calling the result "rogue AI" does more than sensationalize it. It allows those human decisions to disappear quietly from the story.
- nkurz 13d ago
  I think your confidence is likely misplaced, but I vouched for your comment to revive it from the dead because you expressed your opinion clearly and with examples. To the flaggers: flag comments that you think violate the rules, but don't flag something just because you strongly disagree with the claim it makes.
  - LoganDark 12d ago
    It was [flagged] very quickly, too. I checked my screen capture just now, and it happened between 4 and 14 seconds after posting. Either I tripped something automated, or someone with incredible flagging powers was sitting there waiting to immediately suppress my reply. Was a little disheartening to see my supporting evidence hidden for so long.

ozozozd 14d ago

You are not referring to a port scanner, right?

LoganDark 14d ago
I guess. The IPv4 address space is small enough that it's been feasible to scan the entire thing exhaustively for a while now. Before LLMs, a public IPv4 would mostly get automated scripts that try the same things on every address, sometimes depending on what port scans find, yes. But we're going to start seeing more adversaries that have LLMs investigate each individual address to find novel or unique vulnerabilities. You can spray individualized reverse engineering without having to dedicate a real reverse engineer's time to each target. Avoiding the attention of attackers won't be enough to get away with mere security by obscurity.
- tancop 13d ago
  You can use the same RE tools and run your own agents as a defender. Everything they find helps you harden your setup, to the point there might be no exploitable vulns at all after a couple days of intense red teaming. It's not completely bulletproof (unless you do formal verification, another thing LLMs are good at) but the network boundary stops a lot of attack vectors. You can't really do side channel or timing attacks over IP, so all that's left are easier to patch logic bugs.
  - LoganDark 13d ago
    > You can't really do side channel or timing attacks over IP Ehh, this has been disproven dozens of times but that's besides the point, I think. I absolutely agree that defenders should be using every tool at their disposal to harden their infrastructure. A lot of defenders simply don't do that, and that's always been a shame. I truly hope that this increasing threat leads to better defensive effort. People need to realize they can't just get away with it the same as before.