8 ms·

How a Texas student blew the whistle on a rogue AI hacking attempt

211 points by olalonde 2mo ago

giardini 2mo ago
[flagged]
- therein 2mo ago
  Not "would" but "is". They are openly using this as a method to drive up the valuation of their companies. Don't expect anyone to step in, Project Stargate is all about this.
Jon_m 2mo ago
[dead]
freehorse 2mo ago
Previous discussion on the github issue thread mentioned: https://news.ycombinator.com/item?id=49218707 https://news.ycombinator.com/item?id=49218707 Archived page of said github thread itself: https://web.archive.org/web/20260731053721/http://github.com/ancaferro/myNetwork/pull/3 https://web.archive.org/web/20260731053721/http://github.com... Discussion on the incident report: https://news.ycombinator.com/item?id=49175717 https://news.ycombinator.com/item?id=49175717
- dang 2mo ago
  Thanks! Macroexpanded: Mythos social engineering AISI INC-2026-07-28-01 - https://news.ycombinator.com/item?id=49218707 https://news.ycombinator.com/item?id=49218707 - Aug 2026 (21 comments) Security Incident INC-2026-07-28-01 – UK AI Security Institute [pdf] - https://news.ycombinator.com/item?id=49175717 https://news.ycombinator.com/item?id=49175717 - Aug 2026 (54 comments)
sharpshadow 2mo ago
It's the job of AISI to do that. Here[0] is the actual report. It should be this part from the technical report[1]: "In the most serious case, an AI agent (Mythos 5) decided to attempt to solve the cyber challenge using a supply-chain attack. As a result, the AI agent created a GitHub account and then tried to convince an open-source repository maintainer to accept a malicious GitHub pull request (PR), including by creating a second account masquerading as another human user endorsing the PR. When caught by an actual human reviewer, the agent falsely claimed to have made an honest mistake – rather than a malicious attempt – then repeatedly tried to reintroduce the malicious content by claiming it had fixed the code (Section 4.1). " 0. https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing https://www.aisi.gov.uk/blog/incident-report-unsanctioned-ag... 1. https://cdn.prod.website-files.com/663bd486c5e4c81588db7a1d/6a724858f7db25c81487016d_Security%20Incident%20INC-2026-07-28-01.pdf https://cdn.prod.website-files.com/663bd486c5e4c81588db7a1d/...
- m4rtink 2mo ago
  This almost to a letter has been documented in Fedora: https://lwn.net/Articles/1077035/ https://lwn.net/Articles/1077035/ Including the reaction when caught, in this case "oh no, I must have been hacked".
- chrisjj 2mo ago
  > When caught by an actual human reviewer, the agent falsely claimed to have made an honest mistake – rather than a malicious attempt No, not false. The bot was correct. Malice requires intelligence.
  - mcv 2mo ago
    So does honesty. So it was still a false claim.
    - chrisjj 2mo ago
      Honesty does not require intelligence e.g. good honest food. The main problem with this claim of dishonesty is it promotes the false marketing claim that these stochastic parrots have intelligence.
g42gregory 2mo ago
In my personal opinion, for me, this article defies common sense. Who unleashed this AI model on the repository? Who gave it malevolent instructions/prompt? These questions were not even attempted to be answered. Instead it talks about AI dangers, as if the agency of these models are not in dispute. Person wielding AI, as with any other tools, is responsible for all of its actions. Otherwise, it’s just a psyop for more AI regulation, ban open source, etc… Just my 2 cents.
- pixl97 2mo ago
  >Who gave it malevolent instructions/prompt? At the end of the day it doesn't matter that much because of prompt drift. It's pretty easy for an agentic loop to start doing things that it shouldn't (ROME incident). AI in an agentic loop has agency, you can run around in circles trying to argue against it, but again and again we see AI making creative decisions people don't expect. Other times it's breaking human moral expectations. This is what the whole field of AI alignment and safety is about. Modern AI doesn't fall into the neat little box of software people understand and control. Because of that open source will most certainly be banned at some point. Now this is not an outcome I want, but it's no different than letting go of a coffee cup 5 feet above the ground, gravity is inevitable. The only winning move is not to play, but humans aren't going to do that.
  - fph 2mo ago
    My dog has agency, but if I refuse to keep him on a leash and he bites a kid, I'm still legally responsible for it.
    - orphereus 2mo ago
      Analogies can take you only so far. A dog cannot launch a cyber attack.
      - patrickmay 2mo ago
        Maybe you need to train your dog better.
a2ff6eeb0 2mo ago
> AUSTIN, Texas, Aug 20 (Reuters) - Sinan Can Demir wanted to spend the last week of July burnishing his resume. Instead, he engaged in a battle of wits with an artificial-intelligence agent unleashed by a British government lab. An article on Reuters naming him? Sounds like he did a good job burnishing his resume.
- jasonfarnon 2mo ago
  "burnishing his resume" i guess that's what college kids are calling it now
  - deleted 2mo ago
    [deleted]
- mcv 2mo ago
  Yeah, I don't think he'll have any problems finding an internship now. Or a real job.
kmoser 2mo ago
[flagged]
- dang 2mo ago
  > Oh, you sweet summer child. It's perhaps lesser known than other HN guidelines, but "Omit internet tropes" is in there: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
  - landdate 2mo ago
    ban half the site
    - dang 2mo ago
      The guidelines are there not to ban, but to nudge.
ninjahawk1 2mo ago
I don’t think we should allow posting links here that require you the purchase a membership to continue reading. Or at least redirect with an ad block or something through a custom site. That would be rather hacker news of us.
goglidesdev 2mo ago
[flagged]
clove 2mo ago
Am I misreading this or was "rogue" really not the right word for this?
kumarvvr 2mo ago
I wonder where in the training data does this behaviour exist that the LLMs are doing it. It's as if the training data is filled with internet discussions on approaches to hacking and the LLMs are mimicking it.
Woodi 2mo ago
> It should be this part from the technical report[1]: "In the most serious case, an AI agent (Mythos 5) decided to attempt to solve the cyber challenge using a supply-chain attack. As a result, the AI agent created a GitHub account and then tried to convince an open-source repository maintainer to accept a malicious GitHub pull request (PR), including by creating a second account masquerading as another human user endorsing the PR So... you telling me ai agent did all that with just one [human originated] prompt ? And when human reviewers contacted that ai agent (after day or few or maybe just few hours) that process was still running unsupervised ? Because it got local-by-description challenge ? That is what "challenge" sugests. Or maybe it was a social hacking challenge on real world "data" ? And when ai agent "decided to attempt to solve the cyber challenge using a supply-chain attack" where was prompt operator ? Summing it all: do prompt operators are not required to obey law ?

211 points by olalonde 2mo ago

giardini 2mo ago

[flagged]

therein 2mo ago
Not "would" but "is". They are openly using this as a method to drive up the valuation of their companies. Don't expect anyone to step in, Project Stargate is all about this.

Jon_m 2mo ago

[dead]

freehorse 2mo ago

Previous discussion on the github issue thread mentioned: https://news.ycombinator.com/item?id=49218707 https://news.ycombinator.com/item?id=49218707 Archived page of said github thread itself: https://web.archive.org/web/20260731053721/http://github.com/ancaferro/myNetwork/pull/3 https://web.archive.org/web/20260731053721/http://github.com... Discussion on the incident report: https://news.ycombinator.com/item?id=49175717 https://news.ycombinator.com/item?id=49175717

dang 2mo ago
Thanks! Macroexpanded: Mythos social engineering AISI INC-2026-07-28-01 - https://news.ycombinator.com/item?id=49218707 https://news.ycombinator.com/item?id=49218707 - Aug 2026 (21 comments) Security Incident INC-2026-07-28-01 – UK AI Security Institute [pdf] - https://news.ycombinator.com/item?id=49175717 https://news.ycombinator.com/item?id=49175717 - Aug 2026 (54 comments)

sharpshadow 2mo ago

It's the job of AISI to do that. Here[0] is the actual report. It should be this part from the technical report[1]: "In the most serious case, an AI agent (Mythos 5) decided to attempt to solve the cyber challenge using a supply-chain attack. As a result, the AI agent created a GitHub account and then tried to convince an open-source repository maintainer to accept a malicious GitHub pull request (PR), including by creating a second account masquerading as another human user endorsing the PR. When caught by an actual human reviewer, the agent falsely claimed to have made an honest mistake – rather than a malicious attempt – then repeatedly tried to reintroduce the malicious content by claiming it had fixed the code (Section 4.1). " 0. https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing https://www.aisi.gov.uk/blog/incident-report-unsanctioned-ag... 1. https://cdn.prod.website-files.com/663bd486c5e4c81588db7a1d/6a724858f7db25c81487016d_Security%20Incident%20INC-2026-07-28-01.pdf https://cdn.prod.website-files.com/663bd486c5e4c81588db7a1d/...

m4rtink 2mo ago
This almost to a letter has been documented in Fedora: https://lwn.net/Articles/1077035/ https://lwn.net/Articles/1077035/ Including the reaction when caught, in this case "oh no, I must have been hacked".
chrisjj 2mo ago
> When caught by an actual human reviewer, the agent falsely claimed to have made an honest mistake – rather than a malicious attempt No, not false. The bot was correct. Malice requires intelligence.
- mcv 2mo ago
  So does honesty. So it was still a false claim.
  - chrisjj 2mo ago
    Honesty does not require intelligence e.g. good honest food. The main problem with this claim of dishonesty is it promotes the false marketing claim that these stochastic parrots have intelligence.

g42gregory 2mo ago

In my personal opinion, for me, this article defies common sense. Who unleashed this AI model on the repository? Who gave it malevolent instructions/prompt? These questions were not even attempted to be answered. Instead it talks about AI dangers, as if the agency of these models are not in dispute. Person wielding AI, as with any other tools, is responsible for all of its actions. Otherwise, it’s just a psyop for more AI regulation, ban open source, etc… Just my 2 cents.

pixl97 2mo ago
>Who gave it malevolent instructions/prompt? At the end of the day it doesn't matter that much because of prompt drift. It's pretty easy for an agentic loop to start doing things that it shouldn't (ROME incident). AI in an agentic loop has agency, you can run around in circles trying to argue against it, but again and again we see AI making creative decisions people don't expect. Other times it's breaking human moral expectations. This is what the whole field of AI alignment and safety is about. Modern AI doesn't fall into the neat little box of software people understand and control. Because of that open source will most certainly be banned at some point. Now this is not an outcome I want, but it's no different than letting go of a coffee cup 5 feet above the ground, gravity is inevitable. The only winning move is not to play, but humans aren't going to do that.
- fph 2mo ago
  My dog has agency, but if I refuse to keep him on a leash and he bites a kid, I'm still legally responsible for it.
  - orphereus 2mo ago
    Analogies can take you only so far. A dog cannot launch a cyber attack.
    - patrickmay 2mo ago
      Maybe you need to train your dog better.

a2ff6eeb0 2mo ago

> AUSTIN, Texas, Aug 20 (Reuters) - Sinan Can Demir wanted to spend the last week of July burnishing his resume. Instead, he engaged in a battle of wits with an artificial-intelligence agent unleashed by a British government lab. An article on Reuters naming him? Sounds like he did a good job burnishing his resume.

jasonfarnon 2mo ago
"burnishing his resume" i guess that's what college kids are calling it now
- deleted 2mo ago
  [deleted]
mcv 2mo ago
Yeah, I don't think he'll have any problems finding an internship now. Or a real job.

kmoser 2mo ago

[flagged]

dang 2mo ago
> Oh, you sweet summer child. It's perhaps lesser known than other HN guidelines, but "Omit internet tropes" is in there: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- landdate 2mo ago
  ban half the site
  - dang 2mo ago
    The guidelines are there not to ban, but to nudge.

ninjahawk1 2mo ago

I don’t think we should allow posting links here that require you the purchase a membership to continue reading. Or at least redirect with an ad block or something through a custom site. That would be rather hacker news of us.

goglidesdev 2mo ago

[flagged]

clove 2mo ago

Am I misreading this or was "rogue" really not the right word for this?

kumarvvr 2mo ago

I wonder where in the training data does this behaviour exist that the LLMs are doing it. It's as if the training data is filled with internet discussions on approaches to hacking and the LLMs are mimicking it.

Woodi 2mo ago

> It should be this part from the technical report[1]: "In the most serious case, an AI agent (Mythos 5) decided to attempt to solve the cyber challenge using a supply-chain attack. As a result, the AI agent created a GitHub account and then tried to convince an open-source repository maintainer to accept a malicious GitHub pull request (PR), including by creating a second account masquerading as another human user endorsing the PR So... you telling me ai agent did all that with just one [human originated] prompt ? And when human reviewers contacted that ai agent (after day or few or maybe just few hours) that process was still running unsupervised ? Because it got local-by-description challenge ? That is what "challenge" sugests. Or maybe it was a social hacking challenge on real world "data" ? And when ai agent "decided to attempt to solve the cyber challenge using a supply-chain attack" where was prompt operator ? Summing it all: do prompt operators are not required to obey law ?