7 ms·

It is fixable. Model requires proof that you are a legitimate developer of that piece of software. Every Anthropic/OpenAI account will have a list of projects

It is fixable. Model requires proof that you are a legitimate developer of that piece of software. Every Anthropic/OpenAI account will have a list of projects the model is allowed to work on for security issues.

ceejayoz 4mo ago
https://en.wikipedia.org/wiki/XZ_Utils_backdoor https://en.wikipedia.org/wiki/XZ_Utils_backdoor > A subsequent investigation found that the campaign to insert the backdoor into the XZ Utils project was a culmination of over two years of effort, starting in 2021, by a user going by the name "Jia Tan". They used sock puppetry in a pressure campaign against the original maintainer of XZ Utils, eventually being given maintainer permissions on the project.
- dist-epoch 4mo ago
  sure. how many cases like these we had so far? 1, 2? and how long did they work to get commit access?
  - ceejayoz 4mo ago
    > how many cases like these we had so far? As with clever, careful serial killers, it's tough to count the ones we haven't caught.
    - applfanboysbgon 4mo ago
      It's not that tough. You can get an idea by how many people are being murdered. A successful serial killer results in dead people, and a successful infiltration results in malware being executed. If there are no murdered people with unattributed causes of death, or there are no open-source projects with unattributed causes of malware being shipped, you can conclude there are roughly 0 active serial killers / infiltrators. It's possible there are infiltrators who are still working on long-term infiltration and haven't yet attempted to add any malicious code anywhere, but the point is that in terms of actual attempts, we've seen a single one and it wasn't even successful despite years of prep.
      - ceejayoz 4mo ago
        > You can get an idea by how many people are being murdered. No, we can't, as that happens a lot via non-serial killers. A truly successful serial killer is likely one who hides in that noise. No taunting the cops, distributed geographic locations, random methods, avoiding calling cards, and careful not to leave too many traces. It seems likely that some of the 350k unsolved homicides in the US can be explained this way. > It's possible there are infiltrators who are still working on long-term infiltration and haven't yet attempted to add any malicious code anywhere… Or the code's already there, latent, as it would've been in the XZ case, which got discovered by chance and someone very dedicated to looking into a performance glitch.
        deleted 4mo ago
        [deleted]
  - virtualritz 4mo ago
    We only know how many were discovered. Since we do not know the ratio to undiscovered this "1-2" is meaningless to assess the risk of this sort of attack.
- brookst 4mo ago
  Can we retire the “seatbelts are useless because they can’t prevent every loss of life” approach to risk mitigation please? If the acceptance criteria is “would prevent every single past instance and every imaginable future instance”, then yes, no mitigation is every sufficient to address any problem in the world, so we might as well give up. But I don’t think that’s the right lens to use.
  - ceejayoz 4mo ago
    I'm onboard with this! I just object to the term "fixable".
  - pjc50 4mo ago
    That depends on whether it's a issue of accidents or a "you have to get lucky every time, we only have to get lucky once" issue.
    - brookst 4mo ago
      Death only has to get lucky once. Are you going to stop wearing seatbelts?
      - ben_w 4mo ago
        I assume pjc50's quotation is referencing a quote attributed to a terrorist group after they failed to assassinate the UK Prime Minister: https://quoteinvestigator.com/2025/12/08/lucky-always/ https://quoteinvestigator.com/2025/12/08/lucky-always/ You're in control of how much danger of accident you expose yourself to. Nobody is in control of how much danger we are exposed to from other people who are actively trying to do us harm, who will keep going until they get what they're after or are stopped. For most people, seatbelts are the former. Yeah, not perfect, but they reduce risk. For the latter, if you're known to be a seatbelt wearer, the attacker just does something where seatbelts don't matter. Every new AI model introduces new capabilities and competencies, so we're not even sure what the true risk levels are yet for self-exposure in this category. The restrictions on AI may be like seatbelts and speed limits, or they may be like "if you install a 1000 HP turbojet engine in your Honda Civic it will no longer be road legal". And this analogy also includes how the first cars had speed limits set low enough to not risk the horse industry, i.e. we may be too cautious.
_davide_ 4mo ago
Sounds like a good solution my Führer
ReptileMan 4mo ago
Everyone is legitimate developer on open source software...
cogman10 4mo ago
Ok, and how is that determined? How does anthropic know my "kernel" project isn't a personal toy and not the Linux kernel? How does anthropic determine I'm a legitimate kernel hacker? What proof do I give them and how does it tie back to my email? What would the steps be to create a new project? Do I need to send anthropic a list of my team members each time and keep them updated as the company changes? Shall I be giving them access to our company's active directory?
- _fizz_buzz_ 4mo ago
  > How does anthropic know my "kernel" project isn't a personal toy and not the Linux kernel? The Linux Kernel is in its training data. I just tested it. I copied about 20 random lines from the linux kernel and asked which codebase this was from and it could immediately tell.
  - cogman10 4mo ago
    The Linux kernel is also in the free bsd project. I'm allowed to copy as little or as much of the kernel as I like into my personal project thanks to the GPL. Being able to attribute the source of a line of code doesn't help you to know if a repository can be legitimately hacked on. As you could imagine, I might just take all or part of the Linux USB stack from the kernel to retrofit it into my own kernel.
- NiloCK 4mo ago
  This is a credentials and access list oAuth style problem, and not really intractable. For package X, I should be able to present my npm (homebrew, apt, nuget, etc) credentials with publishing rights for the package. If package X is of sufficient public interest (user count, nature/sensitivity of user data, downstream distribution, etc), then the public interest + cryptographic credentials should permit access to best-available security auditing. Yes, we still are trusting trust, that the owner of the package itself is not malicious, but that's not a sharp degradation from status quo.
  - Retr0id 4mo ago
animitronix 4mo ago
lol worst idea ever

ceejayoz 4mo ago

https://en.wikipedia.org/wiki/XZ_Utils_backdoor https://en.wikipedia.org/wiki/XZ_Utils_backdoor > A subsequent investigation found that the campaign to insert the backdoor into the XZ Utils project was a culmination of over two years of effort, starting in 2021, by a user going by the name "Jia Tan". They used sock puppetry in a pressure campaign against the original maintainer of XZ Utils, eventually being given maintainer permissions on the project.

dist-epoch 4mo ago
sure. how many cases like these we had so far? 1, 2? and how long did they work to get commit access?
- ceejayoz 4mo ago
  > how many cases like these we had so far? As with clever, careful serial killers, it's tough to count the ones we haven't caught.
  - applfanboysbgon 4mo ago
    It's not that tough. You can get an idea by how many people are being murdered. A successful serial killer results in dead people, and a successful infiltration results in malware being executed. If there are no murdered people with unattributed causes of death, or there are no open-source projects with unattributed causes of malware being shipped, you can conclude there are roughly 0 active serial killers / infiltrators. It's possible there are infiltrators who are still working on long-term infiltration and haven't yet attempted to add any malicious code anywhere, but the point is that in terms of actual attempts, we've seen a single one and it wasn't even successful despite years of prep.
    - ceejayoz 4mo ago
      > You can get an idea by how many people are being murdered. No, we can't, as that happens a lot via non-serial killers. A truly successful serial killer is likely one who hides in that noise. No taunting the cops, distributed geographic locations, random methods, avoiding calling cards, and careful not to leave too many traces. It seems likely that some of the 350k unsolved homicides in the US can be explained this way. > It's possible there are infiltrators who are still working on long-term infiltration and haven't yet attempted to add any malicious code anywhere… Or the code's already there, latent, as it would've been in the XZ case, which got discovered by chance and someone very dedicated to looking into a performance glitch.
      - deleted 4mo ago
        [deleted]
- virtualritz 4mo ago
  We only know how many were discovered. Since we do not know the ratio to undiscovered this "1-2" is meaningless to assess the risk of this sort of attack.
brookst 4mo ago
Can we retire the “seatbelts are useless because they can’t prevent every loss of life” approach to risk mitigation please? If the acceptance criteria is “would prevent every single past instance and every imaginable future instance”, then yes, no mitigation is every sufficient to address any problem in the world, so we might as well give up. But I don’t think that’s the right lens to use.
- ceejayoz 4mo ago
  I'm onboard with this! I just object to the term "fixable".
- pjc50 4mo ago
  That depends on whether it's a issue of accidents or a "you have to get lucky every time, we only have to get lucky once" issue.
  - brookst 4mo ago
    Death only has to get lucky once. Are you going to stop wearing seatbelts?
    - ben_w 4mo ago
      I assume pjc50's quotation is referencing a quote attributed to a terrorist group after they failed to assassinate the UK Prime Minister: https://quoteinvestigator.com/2025/12/08/lucky-always/ https://quoteinvestigator.com/2025/12/08/lucky-always/ You're in control of how much danger of accident you expose yourself to. Nobody is in control of how much danger we are exposed to from other people who are actively trying to do us harm, who will keep going until they get what they're after or are stopped. For most people, seatbelts are the former. Yeah, not perfect, but they reduce risk. For the latter, if you're known to be a seatbelt wearer, the attacker just does something where seatbelts don't matter. Every new AI model introduces new capabilities and competencies, so we're not even sure what the true risk levels are yet for self-exposure in this category. The restrictions on AI may be like seatbelts and speed limits, or they may be like "if you install a 1000 HP turbojet engine in your Honda Civic it will no longer be road legal". And this analogy also includes how the first cars had speed limits set low enough to not risk the horse industry, i.e. we may be too cautious.

_davide_ 4mo ago

Sounds like a good solution my Führer

ReptileMan 4mo ago

Everyone is legitimate developer on open source software...

cogman10 4mo ago

Ok, and how is that determined? How does anthropic know my "kernel" project isn't a personal toy and not the Linux kernel? How does anthropic determine I'm a legitimate kernel hacker? What proof do I give them and how does it tie back to my email? What would the steps be to create a new project? Do I need to send anthropic a list of my team members each time and keep them updated as the company changes? Shall I be giving them access to our company's active directory?

_fizz_buzz_ 4mo ago
> How does anthropic know my "kernel" project isn't a personal toy and not the Linux kernel? The Linux Kernel is in its training data. I just tested it. I copied about 20 random lines from the linux kernel and asked which codebase this was from and it could immediately tell.
- cogman10 4mo ago
  The Linux kernel is also in the free bsd project. I'm allowed to copy as little or as much of the kernel as I like into my personal project thanks to the GPL. Being able to attribute the source of a line of code doesn't help you to know if a repository can be legitimately hacked on. As you could imagine, I might just take all or part of the Linux USB stack from the kernel to retrofit it into my own kernel.
NiloCK 4mo ago
This is a credentials and access list oAuth style problem, and not really intractable. For package X, I should be able to present my npm (homebrew, apt, nuget, etc) credentials with publishing rights for the package. If package X is of sufficient public interest (user count, nature/sensitivity of user data, downstream distribution, etc), then the public interest + cryptographic credentials should permit access to best-available security auditing. Yes, we still are trusting trust, that the owner of the package itself is not malicious, but that's not a sharp degradation from status quo.
- Retr0id 4mo ago

animitronix 4mo ago

lol worst idea ever