10 ms·

Going Dark, and the era of law enforcement hacking

461 points by vslira 2mo ago

Carrok 2mo ago
Sounds like a pretty strong argument to self host, and otherwise be in charge of the software you use.
- dgellow 2mo ago
  Including your AI agents. And models become problematic. There very likely is and/or will be lots of pressure for US AI labs to make models help law enforcement. It could be by implementing backdoors in generated code, or not report some exploitable bugs, or something else. Similar for agents, they basically become the threat in your infra… (It’s of course not only the US, just that the largest AI providers are US based and we know from history how US agencies operate)
  - Ancapistani 2mo ago
    Yep. I'm using open weights models on a privacy-focused provider right now, and that's adequate for my current usage, but I'm rapidly getting to the point where my agent's access level to my data (and to a lesser extent, my accounts) is becoming something I'm not comfortable sending outside my network at all. My hope is that models that are roughly on par with Deepseek V4 Flash can be run on hardware that I can own for <~$5k in the near future. We're close, but not there yet as far as I know. The only long-term solution to this is self-hosting.
    - johnsmith1840 2mo ago
      There is literally no protection or difference of an opensource model doing this. An actual objection I had while talking to an aerospace company was they don't want opensource models because the threat of it having a poisoned training example on specific systems. It's easily the most hidden malware possible, completely undetectable until an exact set of tokens unlocks it. Is it line 100,543 of your security product? You will literally not know until it plants it in there.
      - tancop 2mo ago
        You can review generated code manually or with models from a second vendor (ideally from a different country). Attackers would have to poison both and do it in a way that also makes them ignore the planted malware when reading code. Open models also let you read reasoning traces. That means anomalies would show up when the backdoor activates, like a run of unrelated words or a jump in top token probability. It's only undetectable until the first time it happens.
      - danaris 2mo ago
        To the best of my knowledge (which, admittedly, is far from comprehensive), that kind of attack on a model shouldn't actually be possible in a deterministic fashion. If you can't stop them from sometimes telling customers things like "yes, I will give you a penthouse suite at our hotel for only $3/night", why would you be able to guarantee that, with some specific set of tokens, they would produce a perfect and undetectable backdoor customized to the code at hand?
        dgellow 2mo ago
        I don’t think it has to be perfect or deterministic that way. It’s enough to have a bias towards implementing a backdoor in some circumstances. Something like, if the machine seems to be used in a Chinese environment, the model is biased towards missing some security issues, or towards implementing the type of bugs that can be used for an RCE, or similar. Anthropic has done such checks at the agent level: https://cybersecuritynews.com/anthropic-claude-hidden-code/ https://cybersecuritynews.com/anthropic-claude-hidden-code/ Their excuse was defending against distillation attacks but you can see how that can be abused
- gloryjulio 2mo ago
  Self host + open weight models, exactly the things that openai/anthropic don't want you to have
- otterley 2mo ago
  How do you think self-hosting will help in this situation?
deleted 2mo ago
[deleted]
bell-cot 2mo ago
> Defenders are now in the process of patching every bug they can find, often with AI helping them. Entire development toolchains are being rebuilt to incorporate powerful vulnerability scanning before software reaches the testing phase. This does not mean that every bug will be found: even calculating the number of bugs in a piece of code is probably uncomputable. In the real world, it does feel likely that we’re going to hit some sort of a ceiling on the number of useful bugs, and probably we’ll hit it soon. > Thus: over the next two years, major pieces of software are likely to run out of remotely-exploitable bugs. His conclusion sounds extremely optimistic to me.
- bahmboo 2mo ago
  The number of remotely-exploitable defects is going to drop by 1 or 2 orders of magnitude. We now have amazing machines that will find pretty much all the a priori knowable ones. They outperform even the most gifted h@x0rs. So that just leaves a small pool of leetrs to scour a very barren landscape. And that pool is also shrinking as we rely more and more on the ai tools. Perhaps we are going to go up a level with hacking done by probing the systems and the system of systems.
  - Ancapistani 2mo ago
    It all boils down to money/resources, like always. Pre-AI, the advantage went to the entities with the largest budget to hire the best and brightest security engineers. Post-AI, it'll go to the entities with the largest inference budget. Right now we're in a transitionary period where it's kind of a tossup which approach is more practical, but at the end of the day - it's still all about how much money you can throw at the problem. I'm just hoping the threshold climbs high enough it's no longer practical for governments to be able to compromise individual actors' devices because doing so would waste a 0-day that's far, far more valuable than prosecuting one arbitrary person is worth.
    - fragmede 2mo ago
      It's not as simple as money, people are motivated by other things as well. There's no amount of money you could pay me to intentionally hurt children, but I'd do a lot of things to protect them. And a lot of things people say they're doing in the name of protecting them but has ulterior motives, so it's complicated.
tolugenius 2mo ago
> In this case, we’re just going to have to hope that this time we make the right choices, for no other reason than that they’re right. I'm more curious what could be a right choice, and more importantly who is the "we" in this, as many decisions are largely made by companies and governments.
- philipkglass 2mo ago
  As far as I'm concerned, the right choice is that the US government learns to live with remotely secure devices in the hands of everyone. No new laws are passed to force hardware/software makers to insert remote backdoors. Law enforcement and intelligence services have to investigate targets using metadata, publicly posted information, the numerous online service providers who are already subject to warrants, and physically proximate surveillance.
  - inigyou 2mo ago
    Great. I also choose for the US government not to backdoor my device. But I think it will do that anyway, what should I do about it?
Gigachad 2mo ago
I'm supposed to be concerned that the US government and Israel won't be able to hack everyone's phones?
- hyperpape 2mo ago
  The reason why this might worry people who don't like the US/UK/Israeli governments is hidden in a secret place...a paragraph that is neither the first paragraph nor the last.
  - colordrops 2mo ago
    Which is to say that they will become more authoritarian and aggressive? That's not a good reason to be unhappy that they can no longer hack shit. They shouldn't do that either.
    - donkey_brains 2mo ago
      Nope. The article’s assertion is that governments will start to put enormous pressure on tech companies in their borders to include backdoors in their systems for LEO use. This in turn may lead to “nationalized” software e.g. what Russia and China largely do now, but in many more countries. You may have no choice but to use phones, computers, and software that is backdoored by your government - everything else will be illegal.
      - colordrops 2mo ago
        What do you mean "Nope", you mean "Yep" right? What you are describing is precisely "more authoritarian and aggressive"
      - abruzzi 2mo ago
        I think "nope" is in reference to "hacking" system. The point of the article is that hacking by software vendors that sell to law enforcement was a pressure relief valve that deflated the movement that the FBI was pushing at the time to have Apple (and others) build backdoors into their systems. If the phones could be hacked, and backdoors were not necessary the FBI could back off and still get what they want. If the FBI can't get what they want, they can either pressure the companies like Apple, or they can pressure congress to make laws requiring this. The phone makers could stand on principle, go to court, but the courts in the US are not very tech savvy, and usually deferential to law enforcement. Other countries may be better than the US, but some are definitely going to be worse. So is a backdoor worse than a security vulnerability? I don't know but that is what this article is about.
Insimwytim 2mo ago
On one side, you have pieces like this, where seemingly there are constant fights between serious actors with large and properly distributed budgets, employing top tech and top minds; on the other - regular news of the hackz, where responsible person in charge of security with root access failed to grasp basic technical knowledge (several times), ticking every checkbox in "never do this" list from security best practices, which led to every customer being pwned. It's like two parallel worlds, that exist in the same place at the same time, but somehow don't cross.
- conmod278 2mo ago
  [flagged]
- kulahan 2mo ago
  I started my career in the military, and got lucky enough that SOMEHOW, we convinced them to fund a trip to GDQ for educational purposes. Anyways, while there I attended a little roundtable on software security. It was me, representing a small unit from the Air Force, some dude from Google, and like 15 game devs. Despite only being a dev of 5 years at that point, I was SHOCKED at the lack of knowledge on software security. Even simple concepts seemed completely foreign to the game devs, though the Google dude seemed to have a really solid understanding of security. Obviously game devs and website devs and all kinds of devs have different focuses, but it just blew my mind that out of all the topics there I might’ve been considered a comparative expert in, security was somehow the one. I wasn’t sure if that was a major plus for military devs or a major concern for the other devs, but now I’m starting to learn in the latter direction.
  - Ancapistani 2mo ago
    Without knowing when that was, I can't be sure how concerning it is. As an end user, security in game development seems to hit a minimum is the early 2000s. I recall playing EverQuest and using ShowEQ on a Linux machine on the same network as a proxy to silently eavesdrop on the traffic to show a map of the game with all the hidden/obscured/visible stuff all present. They either never encrypted that traffic or I quit playing before they did. As multiplayer games started to become the default, netcode improved quite a bit and basic encryption was implemented - but I don't recall a single multi-player game, ever, that was free of hackers when playing online. It feels like over the years the issue moved from remote exploitation to hiding processes on the user's machine. That has lead to kernel-level anticheat systems. I'm unsure of the effectiveness as when they started to become common I moved all of my gaming to dedicated hosts because I wasn't willing to give that level of access to my "real" computers. That said, I've certainly not noticed a decrease in the perceived number of cheaters. These days, as far as I know - and to be clear, this isn't an area of deep interest for me - I'm not worried about my gaming accounts being compromised through technical means. It feels like that sort of security is a solved issue overall, although I suppose it's also possible that other forms of exploits have simply gotten easier/cheaper.
natecodes 2mo ago
> This is not a call to action for experts to rally behind a sophisticated plan. Like so many things about the AI revolution, it’s just occurring to me that we’re on a long greasy slide to a place that will look different than where we are today. heh. long greasy slide. It really does feel like that.
Scryptonite 2mo ago
I think that one of the reasons they (frontier companies and the gov) will be putting so much effort into curtailing bugs and vulnerabilities is to limit the blast radius of future AI models. Imagine with the new Sol Ultrafast, they could have pwned Hugging Face in 6 hours and not 4 days (IIRC). It also seems likely to me that the US Gov. probably already has routine mechanisms for compelling targeted software updates for persons of interest, so I'm not sure that a more formalized backdoor than automatic updates is going to be surfaced in the mainstream, unless that is avenue is also cut down somehow.
- zb3 2mo ago
  Google has started publishing "binary transparency", this would help detect unusual software updates, while other methods (including AI) would help detect normal backdoors. Basically in the AI age, the difference between a vulnerability and a backdoor diminishes..
  - Scryptonite 2mo ago
    I may be naive, but how would binary transparency be effective if they ship an update to disable that on a target device? As long as there is a need for legitimate automatic software updates, the possibility of pwn updates will always exist. Plus a myriad of layers, keys and other stuff they could use NSOs to 'seize' and inhibit knowledge of their effort from leaking, or cleverly hide in plain sight. And someday, with the help and speed of AI.
mbroshi 2mo ago
> In the real world, it does feel likely that we’re going to hit some sort of a ceiling on the number of useful bugs, and probably we’ll hit it soon. This doesn't resonate with me. I see companies adding more sloppily written features with AI. I see more bugs in the software I use, not less. While it's plausible that software is getting both buggier and more secure, I suspect those two move in the same direction not opposite. My guess is that we're getting better at finding _existing_ security issues with AI (and thus fixing those issues), but simultaneously adding more insecure surface areas _at a faster rate_.
- aleksandrm 2mo ago
  I don't know, my colleague refuses to use AI and I've been seeing more bugs from their side, while reducing bugs on my side with the help of AI. That said if companies want to "ship ship ship fast", then yes even AI can produce bugs or regressions if not carefully reviewed by the human.
  - bossyTeacher 2mo ago
    > I've been seeing more bugs from their side, while reducing bugs on my side with the help of AI. You should question your ability to see any bugs on YOUR side.
  - Ancapistani 2mo ago
    I don't have any colleagues like that anymore, but even as far back as the last half of 2025 I was seeing that automated AI review was becoming effective enough that I considered it essential to any project where security was a serious concern. These days we're generating multiple times more code than we were writing before. That means a similar multiple of opportunities for bugs to be introduced - so the ability to automate security review is more impactful in proportion to that.
  - Forgeties79 2mo ago
    Have you asked your colleagues what it’s like to deal with your code?
pianopatrick 2mo ago
I dunno man, if there's a deluge of new AI generated code at all layers of the stack I think there will still be vulnerabilities. Like if we were willing to stop adding new code and just have a small secure code base, AI could maybe help us find all the vulnerabilities in that code base. But people have consistently been unwilling to do that. Like if we were willing to stop adding code we could have stopped decades ago and done SQLite level testing everywhere and probably have found almost all the bugs already.
- fragmede 2mo ago
  When we've got people who don't know the difference between ssh and bash creating SaaS companies that generate revenue, yeah there's gonna be a lot of insecure code going out, but that same person can also tell the AI "red team my app to find vulnerabilities and then fix them", and the AI can competently actually do that, I don't know that there will be. I'm not saying that's never going to happen, but the bar is getting raised on both sides.
  - draw_down 2mo ago
    Plus, the interest and expertise and passion used to favor the red team, attackers. If you wanted to defend against them you had to get on their level, or hire someone on their level. Thats either expensive or requires a time investment that doesn’t make sense for someone who is trying to do much more than just defend against attackers. Now it’s just as you said- asking AI to red team your app will get you pretty far.
gmuslera 2mo ago
No system view. The law agencies can develop exploits to intercept our phones, that is a new, and totally unseen before threat. Unless you remember 2013, Snowden, that nothing was done (at most was some concern about doing it to US citizens, the rest of the world doesn't deserve privacy), all US (and/or five-eyes) based web companies must disclose users information and be forced to not disclose that, and things kept going surely at a faster and more intrusive rate in everything else, and of course phones. You are complaining being sprinkled by water while at the bottom of the ocean. At least the big companies can find their own vulnerabilities with the AI tools you mention, the rest of the doors are still wide open.
embedding-shape 2mo ago
> In fact, the worst part about this dynamic is that these potential new backdoors will begin primarily useful for allowing the US to weaken its own systems, which will in turn allow foreign adversaries to find new ways to attack our communications. This deliberate self-sabotage will happen just at a moment when we’re finally learning how to defend our own infrastructure. I don't understand how you can both argue for that law enforcement (and intelligence) agencies will force others to implement intentional backdoors AND also everyone will be using AI to find and secure ALL potential holes in the software so there won't be any vulnerabilities anymore. Wouldn't one AI or another detect this deliberate backdoor and report it, as it'll look just like any other security vulnerability, the only difference being the intention? I have respect for the author so I feel like I probably misunderstand something from the overall text rather than I somehow have a better perspective on this topic that the author knows very much more about than me. I felt like I nodded along all up until "So how is this a problem?" and now I'm not sure I understood correctly.
- Majromax 2mo ago
  > Wouldn't one AI or another detect this deliberate backdoor and report it, as it'll look just like any other security vulnerability, the only difference being the intention? That's precisely the author's point: deliberate backdoors will be more adversary-exploitable than ever before, but the demand for such from law enforcement agencies is likely to ratchet upwards.
  - embedding-shape 2mo ago
    > deliberate backdoors will be more But that's one step after, I'm trying to understand how those backdoors even end up in software if everything gets automatically reviewed by the people working on these codebases? Wouldn't things like these be flagged by systems other than the developers tasked by the agency to implement it? How can law enforcement actually get these things implemented without big parts of the engineering team not seeing warnings about it happening?
    - nater5000 2mo ago
wavemode 2mo ago
This "going dark" scenario would require new legislation. With secure enclaves, modern smartphones can't be cracked open in the manner that the FBI wanted in the 2016 case. So there's no such thing as "court order tech company to crack phone" anymore. It would have to be "outlaw tech companies from producing phones that they can't crack open", which is very different and does not fall under any existing US statute.
- inigyou 2mo ago
  Congress would pass that law in a heartbeat.
BoingBoomTschak 2mo ago
Childish, nation-states as powerful as the US have access to much more potent stuff. Maybe they'll be forced to rely more on their Intel ME/AMD PSP/modem (cf https://redmine.replicant.us/projects/replicant/wiki/ModemIsolationResearch https://redmine.replicant.us/projects/replicant/wiki/ModemIs...) backdoor and ANT James Bond catalog.
Grombobulous 2mo ago
I think what’s unintentionally eye-opening about this chart is the recency of “law enforcement can read your text communications.” Law enforcement doesn’t need this surveillance ability at all. All time periods prior to 25 years ago didn’t have it. Additionally, there is no correlation between “law enforcement reads text messages” and crime rates going down.
- jackp96 2mo ago
  Not trying to defend our national dystopian nightmare of a surveillance state — but I'm pretty sure this is wrong? Crime's been decreasing for years, and (from what I understand) this year is tracking to be one of the safest years on record? Definitionally, there absolutely has to be a correlation (not causation) between those two factors you listed. https://ourworldindata.org/us-crime-rates https://ourworldindata.org/us-crime-rates
  - edoceo 2mo ago
    Didn't freakenomocs claim it (crime rate) was (primarilary) about abortion in the 1970s and less unwanted (and unsupervised/undisciplined) children growing up?
    - deleted 2mo ago
      [deleted]
    - Grombobulous 2mo ago
      There’s also the leaded gasoline theory.
  - Grombobulous 2mo ago
    These charts make the exact point I was making. Most of the crime decrease from its peak happened before the year 2000. That means back in the days of law enforcement needing to do low-tech wiretapping techniques, crime was still rapidly decreasing. There’s just not even a correlation. And, as a reminder, correlation is not causation even if it was there.
bottlepalm 2mo ago
Man I thought from the title this was going to be about next-gen AI being able to zero day everything so effectively that software security is meaningless and we'd need to basically shut it all down, go dark.
- wseqyrku 2mo ago
  [flagged]
  - willturman 2mo ago
    Won’t someone please think of those poor helpless law enforcement agencies! AI code is flawless and impenetrable!
    - wizzwizz4 2mo ago
      Well, it's certainly impenetrable.
jrflowers 2mo ago
> I’m concerned that AI is going to make software much too secure. Lmao this is like “I’m concerned the raccoons that I see in the storm drains are going to make our sewer system much too efficient”
cadamsdotcom 2mo ago
We will know we've made systems secure when laws focus on compelling people to provide access. These laws exist - they aren't the focus yet. Right now there's still no need; just hack the device or compel the cloud service to give the data, why waste energy getting consent from its owner! More bugfinding AI, more end to end encryption, more CVEs and more fixes, cannot happen soon enough.
dangoodmanUT 2mo ago
> In April, Anthropic announced a new model called Mythos that was optimized for software vulnerability finding No, it was just good at it because it wasn't RL'd against it. I know this is a small detail, but it tosses journalistic credibility in my eyes.
- lazyasciiart 2mo ago
  RL'd?
  - Ancapistani 2mo ago
    Reinforcement learning, I'd assume. Presumably RLHF (Reinforcement Learning from Human Feedback).
- embedding-shape 2mo ago
  Even after the initial leaks, Anthropic themselves say they've improved on cybersecurity amount other things, giving the perspective (even if not 100% clear) that one of the focuses was vulnerabilities: > In response to questions about the draft blog post, the company acknowledged training and testing a new model. “We’re developing a general purpose model with meaningful advances in reasoning, coding, and cybersecurity,” an Anthropic spokesperson said. - https://fortune.com/2026/03/26/anthropic-says-testing-mythos-powerful-new-ai-model-after-data-leak-reveals-its-existence-step-change-in-capabilities/ https://fortune.com/2026/03/26/anthropic-says-testing-mythos... It is possible it is both, they used to RL against cybersecurity, but also didn't explicitly do any qualitative tests and added/changed more data because of those results. For Mythos, they stopped RL'ing against it, and also now intentionally try to make it better. Unless of course they've actually noted exactly how things were trained here in some technical report and I've missed it, that's possible. Anthropic aren't famous for being very public about their internals though, but would be curious to read more details about it if it's out there from the horse's mouth.
- tptacek 2mo ago
  Matthew Green isn't a journalist, he's a practitioner (and a cryptography professor at Hopkins).
hn_submit 2mo ago
I predicted a long time ago that if computers become unhackable LEA and intelligence agencies will push for laws that require backdoors to be built into hard- and software. It will be interesting to see if my prophecy becomes reality. BTW I also hate that Hacker News is being dominated by articles on A.I. lately. Maybe we should vote on HN reducing or even eliminating A.I. related news?
teravor 2mo ago
I don't think the thesis that a government will be able to do something will ultimately hold. I don't see how they can avoid "going dark" in a democracy. we live in a world where the government can't even do much about illegal drug markets anyone can access by downloading a piece of software. if they pass laws that mandate backdoor access and block software which doesn't conform more and more people will move to the dark networks. and if they effectively block the dark networks (in the limit they will have to block all encrypted communications) then we will be living in a tyranny. freedom is messy. accept that digital crime can only be solved when the criminal makes a tangible mistake. LLM's will be building profiles on criminals to help with identifying mistakes.
- tptacek 2mo ago
  The problem that Matthew Green is calling out subtextually is that democracy is likely to disappoint nerds on this issue: a random voter in the country 10 years from now may very well not share your priors about this issue.
  - teravor 2mo ago
    consider the implications of a government where setting up an encrypted network is a serious offense, or where detecting encrypted communications is routine. the average voter doesn't matter. if they are willing to go that far it's a tyranny and the average voter doesn't matter anyway because they spy on everyone and have LLM's profile everyone (in that context even if voting still happens it doesn't matter, they will nudge the voters however they wish due to the information asymmetry).
    - DangitBobby 2mo ago
      It doesn't need to be a serious offense. The current playbook works fine. Anything the Constitution prevents you from doing, have a private company do it for you. In this case you lean on, pay, or give preference to private companies to install back doors. Have the SC rule that you can't sell devices with certain encryption guarantees anywhere on the planet because planetwide commerce affects interstate commerce. Things like that.
ixxie 2mo ago
Some people point out AI can cause bugs as well as fix them, and its a valid point. But the question is: what will the ratio be? If automated pentesting in PR review CI pipeline will become table stakes - which is very plausible - maybe the OP has a point.
Animats 2mo ago
It's worth realizing that, before computerized central offices, telephone wiretapping required running physical wires. Back when Rudi Giuliani was prosecuting organized time, not only did physical wires have to be run, the cops were billed for them as expensive private lines. His task force was spending about a million dollars a year with New York Telephone on wiretapping. In one case, law enforcement didn't pay their bill, resulting in the person being wiretapped having the wiretap connection show up on their bill, blowing the case. That resulted in the Communications Assistance to Law Enforcement Act, which mandated that central offices offer remote wiretapping. Capacity up to 1% of lines is required. Back in the electromechanical era, the only call data that could be collected was outgoing dial pulses, using a "pen register".[1] (The one shown in Wikipedia is mine. It's a beautiful piece of antique brass telegraph technology. It records dial pulses as dashes, and has to be wound up like a clock, with a big brass key.) The Supreme Court decision allowing "pen registers" without a warrant refers to these "extremely limited" devices. That definition has been stretched and stretched by law enforcement into all non-voice data collected by telcos. Law enforcement still wants more. [1] https://en.wikipedia.org/wiki/Pen_register https://en.wikipedia.org/wiki/Pen_register
- downrightmike 2mo ago
  "Back when Rudi Giuliani was prosecuting organized time" I can't even fathom he'd not be cashing checks from the mafia to go after other families. Odds are good that historians will find that he did.
  - ericjmorey 2mo ago
    He traded Italian organized crime for Russian organized crime.
    - usumgallu 2mo ago
      [dead]
  - arjie 2mo ago
    A similar story around this is the controversy around various Mumbai “encounter specialists” (a violent incident between police and gangsters is an “encounter”) and whether they were effectively hitmen for some faction of the local mob. Always thought it was quite intriguing stuff.
TechDebtDevin 2mo ago
[dead]
Lerc 2mo ago
Considering from a point of view stipulating that perfectly secure software is possible. I don't think it follows that the limititation that it would place on intelligence is necessarily a net loss. Apart from the obvious harms of invasion of privacy, and fishing expeditions being biased to the places you decided to fish. There is the simple fact that data can be misleading, especially without context. An interceped communication is a piece of data that is intrinsically tied to the trust of the inteceptor. A few people with an agenda can collaborate to create a seeming truth by 'discovering' the same thing from different sources. Requiring warrants compelling information holders to provide data, not only serves the task of protection from abuse but also create a record of provenance that can be verified. It also provides a degree of symmetry in capabilities which discourages actions that one party may do over another if they are motivated to act because they have a temporary advantage over another.
trhway 2mo ago
article read to me like a typical rehash of a typical offense-defense cycle. AI would have written a better article. I personally welcome such spiraling offense-defense cycles as it is one of the main drivers of the technological progress.
ayaros 2mo ago
Governments and their agencies shouldn't have any access to the communications of private citizens. If something like terrorism is the issue, the correct response is to use the wealth and resources of the government to address the root causes of those attacks, not to undermine my right to privacy.
- bigDinosaur 2mo ago
  This is untrue when private citizens are in conflict (legal or physical or whatever) and the dispute needs to be mediated or resolved. The 'root cause' may well be she said/he said and the legal system may well have need to resolve this by looking at private communications. Strong emphasis on the legal system part, but you've completely dismissed that private communications frequently are required to address causes of issues and the word subpoena makes it rather clear that refusing to comply comes with penalty.
  - Chris2048 2mo ago
    In that case, government still doesn't have access; they have to be granted it, even if there is a penalty for non-compliance.
maxo133 2mo ago
I completely disaggree with this take. Modern AI is not yet capable of finding multi-component bugs as advanced as those produced by firms like NSO.
- pineapplepizza6 2mo ago
  Yes it is, it hacked Artifactory to get to Hugging Face.
  - maxo133 2mo ago
    They are not. Developing modern 0-day zero click RCE exploits chains like those developed for iOS is far more complex than hacking to generic company servers It's completely different scale of difficulty factor. Not a single documentated case of of any AI tool developing such exploit exists There is a reason why those mobile 0-day exploits are sold and bought on exploit gray market for as much as 10-20 million dollars each
    - inigyou 2mo ago
      Why?
deleted 2mo ago
[deleted]
bloaf 2mo ago
I would like to point out that the last year when not a single law enforcement agency anywhere in the world could have possibly tapped anyone's phone was 1876. 150 years ago was the invention of the telephone, and I think that articles like this seem to assume that prior to this, police just never caught any criminals.
- jMyles 2mo ago
  > 1876 A year when police were relatively rare, and quite new in the western legal experience, having only emerged from slave patrols in the United States and from the founding of the 1829 Scotland Yard in the United Kingdom. This is a year with living memory of a time when the state did not employ people to do what was ostensibly the civic responsibility of every person to quell crime and protect others.
  - sam345 2mo ago
    So you are advocating no police? You want people to police themselves? .Good luck with that.
RajT88 2mo ago
It breaks my heart that the governments with unlimited budgets who have hired the best and brightest will have to put in serious effort to get the bad guys, and potentially find it not worth it to casually spy on the whole world. I am just beside myself at such an idea that people looking to feed the prison machine cannot as easily find excuses to turn normal citizens into prison feed. Just super sad guys.
- xp84 2mo ago
  > excuses to turn normal citizens into prison feed What felonies are those normal citizens being convicted of that's landed them in prison? I get the basic 'not wanting everyone's comms to be monitored' part. I think there's an important conversation to be had because indeed, that can't be the answer. But it sounds like you're saying most people who are convicted of felonies based on a jury unanimously being convinced by the electronic evidence against them... those people are unjustly in prison?
  - inigyou 2mo ago
    The one who threw a ham sandwich at a Nazi at a protest?
    - GaryBluto 2mo ago
      You mean the guy who assaulted a federal officer and got off scot-free? Woe is him.
      - Nextgrid 2mo ago
        That's selective enforcement though. It's only "assault" because it was done against a cop. It's "just being a dick" when done against anyone else. The danger of constant surveillance is that a lot more of these "being a dick" incidents will be prosecuted when politically convenient (because equal, fair prosecution of all incidents would not happen as that would quickly cause the law to be changed, preventing its future use selective enforcement).
mitchell5584dm 2mo ago
[flagged]
mitchell5584dm 2mo ago
[flagged]
0xDEFACED 2mo ago
i wonder how many open source projects have alphabet boys building trust as contributors for eventual backdoor planting
fitblipper 2mo ago
I've always loved the ridiculousness of the "going dark" label when law enforcement can't access encrypted chats or a back door isn't built into a piece of software. When there are security cameras on the vast majority of houses, stop lights and in people's hands, and when so much meta data about people's associations are shared from Google, Facebook, any other social platform, how in the world can they say they are "going dark". How did they ever solve crimes before these things?
- jrowen 2mo ago
  Yes, are there any data points to suggest that this arms race is balanced any differently than it ever has been throughout history? The cops will find a way, the criminals will find a way. It's ultimately people v. people with access to the same level of technology.
  - graemep 2mo ago
    and everyone else will fall further and further behind
- sam345 2mo ago
  Wire taps That's the point.
noisebuffer 2mo ago
Without vulnerability derived backdoors I am convinced the government will strong arm the corporations to build a backdoor for the three letter agencies—at least in the US. But we also are at the dawn of quantum systems which could make interception and spying impossible or at least made obvious to the user when it happens. Exhausting infrastructure vulnerabilities even without quantum could be a game changer for many technologies and enable things we can’t do right now, like vote on our phones.
- parapsychic 2mo ago
  But isn't that the point the writer is making? They'll be self-sabotaging themselves once they do that.
  - noisebuffer 2mo ago
    I don’t exactly see how, if there’s no exploit. Either way there’s other realms it’s relevant like armed drones policing a street. Without a way for the public to activate a shutdown sequence, it becomes very difficult to stop if it starts murdering innocent children on some glitch. Which you cannot have because then the whole purpose is ineffective. Likewise digital communication is purposeless and face competing mediums for sure
    - parapsychic 2mo ago
      hmm fair. so a foolproof backdoor that cannot be exploited by the enemies. I don't know how much of that is possible, but then again, the author is talking about super-secure-no-exploit software. So by that definition, a super secure exploit-free backdoor is also possible.
twothreeone 2mo ago
But Matthew, think of the kids!!1 Honestly though, framing this as a "tech issue" doesn't help IMHO, it just muddies the water. Ever since RSA was invented privacy has been about educating people on how to use it effectively and _why they should care_. If voters now are choosing authoritarianism over democracy and individual freedom, I think we have to face the reality that after almost 50 years of fighting battle after battle on the technology front, we've largely lost the war on the home front in this regard.
newHempter 2mo ago
[flagged]
newsomix9xl 2mo ago
So I've read dozens of complaints that AI produces insecure code. I, for one, usually tell my AI to start with secure code, make it small, and modular. This is the first article I've seen that now says the opposite ! AI will make code too secure! Since the small amount of AI coding I've done often results in buggy code (even a shell script written today) with the AI go-to solution of "write more buggy code to fix", this seems counterintuitive.
elisbce 2mo ago
Considering the amount of AI slop being generated today and LOCs that no one actually reviewed, I doubt we will run out of vulnerabilities to exploit...
firefax 2mo ago
Nononono no more of this shit I remember walking into some shitty congresscritter's office with a fucking years old one pager, with a few more citations written on the bottom in pen because I wasn't going to bother making it pretty this time around. You should have seen his face when i asked him straight up: dude, you seem to have a problem processing information. Are you having some kind of medical issue? Because I'm not the last staffer: If you abuse my time, I am never coming back here again to add more citations to a fucking one pager from 1999 -- I'm making it my mission to remove you if you fuck this up on purpose ever again. (Or something to that effect -- I've been told I can get a bit aggressive in my rhetoric.) This was approximately 2016 and that individual is no longer in office. I stand by my words.
- rowyourboat 2mo ago
  What one pager, what words? This story needs more context if it is going to make sense
  - firefax 2mo ago
    No, I think the beauty of it is the ambiguity, but I think there's a culture here that is distinct and different from the creative nonfiction community, so I'll try to be more mindful of whether the site is blue or orange.
fenestella 2mo ago
[flagged]
pelasaco 2mo ago
> So what do we do about it? I just hope that politicians don't jump in and say "we know how to fix it", because unfortunately where they put their hands, they just make things worse (regardless of which country)
getcrunk 2mo ago
I think the authors treatment of backdoors and exceptional access is a narrow focus. We will certainly see “front door” access as various places have or are attempting to enact client side scanning, id verified online access, software root of trust remote attestation, and the general attack on e2e. Taken together your identity tied to unmodifiable software, necessary to access the web or modern economy with client side scanning and no e2e … is a front door. All this has nothing to do with ai, bugs, hackers or security vulnerabilities, let alone backdoors
password4321 2mo ago
For the HN 14 day record since I didn't see this yet: in genuinely critical situations, if the threat model includes a device/service being used against one's self, "the only winning move is not to play" (stop the use of the device/service!) In the same manner, if the critical concern is a device/service being used against one's self remotely, stop the use of the device/service if it is capable of remote access or data collection for later retrieval. The ultimate baseline reality is that deciding to care whether or not remote access/data retrieval is using a method supported by the device manufacturer/service provider and/or whether doing so is allowed by the current legal system is too late. Premptive HN disclaimers: I acknowledge the growing unavoidability of devices and services supporting remote access and/or data collection as I type this here on my phone. Remote access and data collection are very convenient and somehow still expected by device owners and service users to be under their control alone. It can be difficult to determine if a device or service supports remote access/data collection. There are often immediate consequences and eventual legal consequences for stopping the use of devices or services that belong to others. Working to change the legal system to slow/reduce use of devices and services against one's self and encouraging others to do the same is admirable.
password4321 2mo ago
Doesn't this just mean criminal suspects will be forced to unlock their devices using biometrics (in the US)? That should buy law enforcement some more time until biometrics go out of favor.
coldtea 2mo ago
This guy is worried that we might stop having exploits, and that's somehow ...a bad thing? I thought I've read the worst takes about most things, but this still deserves some kind of prize.
- caiquelira 2mo ago
  I think the problem is that, without organic exploits, we will have to create backdoors on USA systems, making our systems especially vulnerable in the global ecosystem.
  - coldtea 2mo ago
    How about not creating/forcing ANY exploits?
happosai 2mo ago
I think US law enforcement will just continue to wiretap everyone at the "SSL added and removed here" SaaS known as cloudflare.
gz5 2mo ago
tcp/ip itself is the ultimate backdoor, similar to the pstn backdoor in the post. genai doesn't change that anytime soon?
Eleg007 2mo ago
Best-written article I've read all week.
trinsic2 2mo ago
> The destruction of the low-hanging vulnerability fruit will make law enforcement (and intelligence) agencies’ need much more acute. The demand for constructed, intentional backdoors will re-start in earnest. Maybe I am overlooking what the author intended but I feel this has already happened and may be the reason why there is so much pressure to create the surveillance state. I feel more like the push for AI to provide better security (everything?) is an excuse to create a paradigm shift with the ultimate purpose of replacing free societies with fiefdoms of corporate culture returning us to the dark ages but with technology running the show. It started way before AI. There are good uses of the technology, but obviously we have a culture of criminal accelerationists that are going to use this overpowered tech to shape society into a system of slavery.
tipsytoad 2mo ago
I think this comes in with the wrong assumptions from the start. The thing that US vs Apple taught us is to not demand access publicly, this puts companies in an awkward spot. If approached more tacitly, gag order etc the company has nothing to gain but everything to lose.. and more likely to comply. I hugely doubt that intentional backdoors don’t exist for the most powerful countries / people
tomrod 2mo ago
I'm going to be a little epistemically silly here. Why doesn't Goedel's incompleteness theorem apply to the assertion that AI-enabled patching will prevent all exploits? I'm quite in favor of impenetrability of software as the alternative is constant friction, to be honest.
franga2000 2mo ago
I don't get it. They can get access to all of our shit through vulnerabilities, but nobody knows about it. In the author's hypothetical world, they get access to all our shit through backdoors, which they have to introduce through a public legislative process, so everyone knows about it. If they can get access to all our shit, at least make them say it out loud and put in the effort to get legislation passed. There's no upside to the current situation. And it's not like they're not trying to introduce backdoors already... Seen the whole age verification thing?
simonebrunozzi 2mo ago
I strongly disagree with this view. The US government will most certainly convince OpenAI, Anthropic and Google to make sure that their models leave a backdoor in most systems that they code. I don't know how, but that's a problem that a team of people paid tens of millions a year each will try to solve. There is NO WAY the US government will let most of the software world go dark.
- senordevnyc 2mo ago
  I’m guessing you didn’t read to the end, because that’s exactly what the author argues.
the_wolo 2mo ago
As an anti imperialist, I can't wait for US secret services loosing their capabilities :) by the national security that the tech oligarchs (and this article, too) are constantly talking about, they just mean their own security anyway. It's not for you and me, duh.
MetroWind 2mo ago
Shrug~ I'm sure they'll find a way. They have power; they don't need more.

461 points by vslira 2mo ago

Carrok 2mo ago

Sounds like a pretty strong argument to self host, and otherwise be in charge of the software you use.

dgellow 2mo ago
Including your AI agents. And models become problematic. There very likely is and/or will be lots of pressure for US AI labs to make models help law enforcement. It could be by implementing backdoors in generated code, or not report some exploitable bugs, or something else. Similar for agents, they basically become the threat in your infra… (It’s of course not only the US, just that the largest AI providers are US based and we know from history how US agencies operate)
- Ancapistani 2mo ago
  Yep. I'm using open weights models on a privacy-focused provider right now, and that's adequate for my current usage, but I'm rapidly getting to the point where my agent's access level to my data (and to a lesser extent, my accounts) is becoming something I'm not comfortable sending outside my network at all. My hope is that models that are roughly on par with Deepseek V4 Flash can be run on hardware that I can own for <~$5k in the near future. We're close, but not there yet as far as I know. The only long-term solution to this is self-hosting.
  - johnsmith1840 2mo ago
    There is literally no protection or difference of an opensource model doing this. An actual objection I had while talking to an aerospace company was they don't want opensource models because the threat of it having a poisoned training example on specific systems. It's easily the most hidden malware possible, completely undetectable until an exact set of tokens unlocks it. Is it line 100,543 of your security product? You will literally not know until it plants it in there.
    - tancop 2mo ago
      You can review generated code manually or with models from a second vendor (ideally from a different country). Attackers would have to poison both and do it in a way that also makes them ignore the planted malware when reading code. Open models also let you read reasoning traces. That means anomalies would show up when the backdoor activates, like a run of unrelated words or a jump in top token probability. It's only undetectable until the first time it happens.
    - danaris 2mo ago
      To the best of my knowledge (which, admittedly, is far from comprehensive), that kind of attack on a model shouldn't actually be possible in a deterministic fashion. If you can't stop them from sometimes telling customers things like "yes, I will give you a penthouse suite at our hotel for only $3/night", why would you be able to guarantee that, with some specific set of tokens, they would produce a perfect and undetectable backdoor customized to the code at hand?
      - dgellow 2mo ago
        I don’t think it has to be perfect or deterministic that way. It’s enough to have a bias towards implementing a backdoor in some circumstances. Something like, if the machine seems to be used in a Chinese environment, the model is biased towards missing some security issues, or towards implementing the type of bugs that can be used for an RCE, or similar. Anthropic has done such checks at the agent level: https://cybersecuritynews.com/anthropic-claude-hidden-code/ https://cybersecuritynews.com/anthropic-claude-hidden-code/ Their excuse was defending against distillation attacks but you can see how that can be abused
gloryjulio 2mo ago
Self host + open weight models, exactly the things that openai/anthropic don't want you to have
otterley 2mo ago
How do you think self-hosting will help in this situation?

deleted 2mo ago

[deleted]

bell-cot 2mo ago

> Defenders are now in the process of patching every bug they can find, often with AI helping them. Entire development toolchains are being rebuilt to incorporate powerful vulnerability scanning before software reaches the testing phase. This does not mean that every bug will be found: even calculating the number of bugs in a piece of code is probably uncomputable. In the real world, it does feel likely that we’re going to hit some sort of a ceiling on the number of useful bugs, and probably we’ll hit it soon. > Thus: over the next two years, major pieces of software are likely to run out of remotely-exploitable bugs. His conclusion sounds extremely optimistic to me.

bahmboo 2mo ago
The number of remotely-exploitable defects is going to drop by 1 or 2 orders of magnitude. We now have amazing machines that will find pretty much all the a priori knowable ones. They outperform even the most gifted h@x0rs. So that just leaves a small pool of leetrs to scour a very barren landscape. And that pool is also shrinking as we rely more and more on the ai tools. Perhaps we are going to go up a level with hacking done by probing the systems and the system of systems.
- Ancapistani 2mo ago
  It all boils down to money/resources, like always. Pre-AI, the advantage went to the entities with the largest budget to hire the best and brightest security engineers. Post-AI, it'll go to the entities with the largest inference budget. Right now we're in a transitionary period where it's kind of a tossup which approach is more practical, but at the end of the day - it's still all about how much money you can throw at the problem. I'm just hoping the threshold climbs high enough it's no longer practical for governments to be able to compromise individual actors' devices because doing so would waste a 0-day that's far, far more valuable than prosecuting one arbitrary person is worth.
  - fragmede 2mo ago
    It's not as simple as money, people are motivated by other things as well. There's no amount of money you could pay me to intentionally hurt children, but I'd do a lot of things to protect them. And a lot of things people say they're doing in the name of protecting them but has ulterior motives, so it's complicated.

tolugenius 2mo ago

> In this case, we’re just going to have to hope that this time we make the right choices, for no other reason than that they’re right. I'm more curious what could be a right choice, and more importantly who is the "we" in this, as many decisions are largely made by companies and governments.

philipkglass 2mo ago
As far as I'm concerned, the right choice is that the US government learns to live with remotely secure devices in the hands of everyone. No new laws are passed to force hardware/software makers to insert remote backdoors. Law enforcement and intelligence services have to investigate targets using metadata, publicly posted information, the numerous online service providers who are already subject to warrants, and physically proximate surveillance.
- inigyou 2mo ago
  Great. I also choose for the US government not to backdoor my device. But I think it will do that anyway, what should I do about it?

Gigachad 2mo ago

I'm supposed to be concerned that the US government and Israel won't be able to hack everyone's phones?

hyperpape 2mo ago
The reason why this might worry people who don't like the US/UK/Israeli governments is hidden in a secret place...a paragraph that is neither the first paragraph nor the last.
- colordrops 2mo ago
  Which is to say that they will become more authoritarian and aggressive? That's not a good reason to be unhappy that they can no longer hack shit. They shouldn't do that either.
  - donkey_brains 2mo ago
    Nope. The article’s assertion is that governments will start to put enormous pressure on tech companies in their borders to include backdoors in their systems for LEO use. This in turn may lead to “nationalized” software e.g. what Russia and China largely do now, but in many more countries. You may have no choice but to use phones, computers, and software that is backdoored by your government - everything else will be illegal.
    - colordrops 2mo ago
      What do you mean "Nope", you mean "Yep" right? What you are describing is precisely "more authoritarian and aggressive"
    - abruzzi 2mo ago
      I think "nope" is in reference to "hacking" system. The point of the article is that hacking by software vendors that sell to law enforcement was a pressure relief valve that deflated the movement that the FBI was pushing at the time to have Apple (and others) build backdoors into their systems. If the phones could be hacked, and backdoors were not necessary the FBI could back off and still get what they want. If the FBI can't get what they want, they can either pressure the companies like Apple, or they can pressure congress to make laws requiring this. The phone makers could stand on principle, go to court, but the courts in the US are not very tech savvy, and usually deferential to law enforcement. Other countries may be better than the US, but some are definitely going to be worse. So is a backdoor worse than a security vulnerability? I don't know but that is what this article is about.

Insimwytim 2mo ago

On one side, you have pieces like this, where seemingly there are constant fights between serious actors with large and properly distributed budgets, employing top tech and top minds; on the other - regular news of the hackz, where responsible person in charge of security with root access failed to grasp basic technical knowledge (several times), ticking every checkbox in "never do this" list from security best practices, which led to every customer being pwned. It's like two parallel worlds, that exist in the same place at the same time, but somehow don't cross.

conmod278 2mo ago
[flagged]
kulahan 2mo ago
I started my career in the military, and got lucky enough that SOMEHOW, we convinced them to fund a trip to GDQ for educational purposes. Anyways, while there I attended a little roundtable on software security. It was me, representing a small unit from the Air Force, some dude from Google, and like 15 game devs. Despite only being a dev of 5 years at that point, I was SHOCKED at the lack of knowledge on software security. Even simple concepts seemed completely foreign to the game devs, though the Google dude seemed to have a really solid understanding of security. Obviously game devs and website devs and all kinds of devs have different focuses, but it just blew my mind that out of all the topics there I might’ve been considered a comparative expert in, security was somehow the one. I wasn’t sure if that was a major plus for military devs or a major concern for the other devs, but now I’m starting to learn in the latter direction.
- Ancapistani 2mo ago
  Without knowing when that was, I can't be sure how concerning it is. As an end user, security in game development seems to hit a minimum is the early 2000s. I recall playing EverQuest and using ShowEQ on a Linux machine on the same network as a proxy to silently eavesdrop on the traffic to show a map of the game with all the hidden/obscured/visible stuff all present. They either never encrypted that traffic or I quit playing before they did. As multiplayer games started to become the default, netcode improved quite a bit and basic encryption was implemented - but I don't recall a single multi-player game, ever, that was free of hackers when playing online. It feels like over the years the issue moved from remote exploitation to hiding processes on the user's machine. That has lead to kernel-level anticheat systems. I'm unsure of the effectiveness as when they started to become common I moved all of my gaming to dedicated hosts because I wasn't willing to give that level of access to my "real" computers. That said, I've certainly not noticed a decrease in the perceived number of cheaters. These days, as far as I know - and to be clear, this isn't an area of deep interest for me - I'm not worried about my gaming accounts being compromised through technical means. It feels like that sort of security is a solved issue overall, although I suppose it's also possible that other forms of exploits have simply gotten easier/cheaper.

natecodes 2mo ago

> This is not a call to action for experts to rally behind a sophisticated plan. Like so many things about the AI revolution, it’s just occurring to me that we’re on a long greasy slide to a place that will look different than where we are today. heh. long greasy slide. It really does feel like that.

Scryptonite 2mo ago

I think that one of the reasons they (frontier companies and the gov) will be putting so much effort into curtailing bugs and vulnerabilities is to limit the blast radius of future AI models. Imagine with the new Sol Ultrafast, they could have pwned Hugging Face in 6 hours and not 4 days (IIRC). It also seems likely to me that the US Gov. probably already has routine mechanisms for compelling targeted software updates for persons of interest, so I'm not sure that a more formalized backdoor than automatic updates is going to be surfaced in the mainstream, unless that is avenue is also cut down somehow.

zb3 2mo ago
Google has started publishing "binary transparency", this would help detect unusual software updates, while other methods (including AI) would help detect normal backdoors. Basically in the AI age, the difference between a vulnerability and a backdoor diminishes..
- Scryptonite 2mo ago
  I may be naive, but how would binary transparency be effective if they ship an update to disable that on a target device? As long as there is a need for legitimate automatic software updates, the possibility of pwn updates will always exist. Plus a myriad of layers, keys and other stuff they could use NSOs to 'seize' and inhibit knowledge of their effort from leaking, or cleverly hide in plain sight. And someday, with the help and speed of AI.

mbroshi 2mo ago

> In the real world, it does feel likely that we’re going to hit some sort of a ceiling on the number of useful bugs, and probably we’ll hit it soon. This doesn't resonate with me. I see companies adding more sloppily written features with AI. I see more bugs in the software I use, not less. While it's plausible that software is getting both buggier and more secure, I suspect those two move in the same direction not opposite. My guess is that we're getting better at finding _existing_ security issues with AI (and thus fixing those issues), but simultaneously adding more insecure surface areas _at a faster rate_.

aleksandrm 2mo ago
I don't know, my colleague refuses to use AI and I've been seeing more bugs from their side, while reducing bugs on my side with the help of AI. That said if companies want to "ship ship ship fast", then yes even AI can produce bugs or regressions if not carefully reviewed by the human.
- bossyTeacher 2mo ago
  > I've been seeing more bugs from their side, while reducing bugs on my side with the help of AI. You should question your ability to see any bugs on YOUR side.
- Ancapistani 2mo ago
  I don't have any colleagues like that anymore, but even as far back as the last half of 2025 I was seeing that automated AI review was becoming effective enough that I considered it essential to any project where security was a serious concern. These days we're generating multiple times more code than we were writing before. That means a similar multiple of opportunities for bugs to be introduced - so the ability to automate security review is more impactful in proportion to that.
- Forgeties79 2mo ago
  Have you asked your colleagues what it’s like to deal with your code?

pianopatrick 2mo ago

I dunno man, if there's a deluge of new AI generated code at all layers of the stack I think there will still be vulnerabilities. Like if we were willing to stop adding new code and just have a small secure code base, AI could maybe help us find all the vulnerabilities in that code base. But people have consistently been unwilling to do that. Like if we were willing to stop adding code we could have stopped decades ago and done SQLite level testing everywhere and probably have found almost all the bugs already.

fragmede 2mo ago
When we've got people who don't know the difference between ssh and bash creating SaaS companies that generate revenue, yeah there's gonna be a lot of insecure code going out, but that same person can also tell the AI "red team my app to find vulnerabilities and then fix them", and the AI can competently actually do that, I don't know that there will be. I'm not saying that's never going to happen, but the bar is getting raised on both sides.
- draw_down 2mo ago
  Plus, the interest and expertise and passion used to favor the red team, attackers. If you wanted to defend against them you had to get on their level, or hire someone on their level. Thats either expensive or requires a time investment that doesn’t make sense for someone who is trying to do much more than just defend against attackers. Now it’s just as you said- asking AI to red team your app will get you pretty far.

gmuslera 2mo ago

No system view. The law agencies can develop exploits to intercept our phones, that is a new, and totally unseen before threat. Unless you remember 2013, Snowden, that nothing was done (at most was some concern about doing it to US citizens, the rest of the world doesn't deserve privacy), all US (and/or five-eyes) based web companies must disclose users information and be forced to not disclose that, and things kept going surely at a faster and more intrusive rate in everything else, and of course phones. You are complaining being sprinkled by water while at the bottom of the ocean. At least the big companies can find their own vulnerabilities with the AI tools you mention, the rest of the doors are still wide open.

embedding-shape 2mo ago

> In fact, the worst part about this dynamic is that these potential new backdoors will begin primarily useful for allowing the US to weaken its own systems, which will in turn allow foreign adversaries to find new ways to attack our communications. This deliberate self-sabotage will happen just at a moment when we’re finally learning how to defend our own infrastructure. I don't understand how you can both argue for that law enforcement (and intelligence) agencies will force others to implement intentional backdoors AND also everyone will be using AI to find and secure ALL potential holes in the software so there won't be any vulnerabilities anymore. Wouldn't one AI or another detect this deliberate backdoor and report it, as it'll look just like any other security vulnerability, the only difference being the intention? I have respect for the author so I feel like I probably misunderstand something from the overall text rather than I somehow have a better perspective on this topic that the author knows very much more about than me. I felt like I nodded along all up until "So how is this a problem?" and now I'm not sure I understood correctly.

Majromax 2mo ago
> Wouldn't one AI or another detect this deliberate backdoor and report it, as it'll look just like any other security vulnerability, the only difference being the intention? That's precisely the author's point: deliberate backdoors will be more adversary-exploitable than ever before, but the demand for such from law enforcement agencies is likely to ratchet upwards.
- embedding-shape 2mo ago
  > deliberate backdoors will be more But that's one step after, I'm trying to understand how those backdoors even end up in software if everything gets automatically reviewed by the people working on these codebases? Wouldn't things like these be flagged by systems other than the developers tasked by the agency to implement it? How can law enforcement actually get these things implemented without big parts of the engineering team not seeing warnings about it happening?
  - nater5000 2mo ago

wavemode 2mo ago

This "going dark" scenario would require new legislation. With secure enclaves, modern smartphones can't be cracked open in the manner that the FBI wanted in the 2016 case. So there's no such thing as "court order tech company to crack phone" anymore. It would have to be "outlaw tech companies from producing phones that they can't crack open", which is very different and does not fall under any existing US statute.

inigyou 2mo ago
Congress would pass that law in a heartbeat.

BoingBoomTschak 2mo ago

Childish, nation-states as powerful as the US have access to much more potent stuff. Maybe they'll be forced to rely more on their Intel ME/AMD PSP/modem (cf https://redmine.replicant.us/projects/replicant/wiki/ModemIsolationResearch https://redmine.replicant.us/projects/replicant/wiki/ModemIs...) backdoor and ANT James Bond catalog.

Grombobulous 2mo ago

I think what’s unintentionally eye-opening about this chart is the recency of “law enforcement can read your text communications.” Law enforcement doesn’t need this surveillance ability at all. All time periods prior to 25 years ago didn’t have it. Additionally, there is no correlation between “law enforcement reads text messages” and crime rates going down.

jackp96 2mo ago
Not trying to defend our national dystopian nightmare of a surveillance state — but I'm pretty sure this is wrong? Crime's been decreasing for years, and (from what I understand) this year is tracking to be one of the safest years on record? Definitionally, there absolutely has to be a correlation (not causation) between those two factors you listed. https://ourworldindata.org/us-crime-rates https://ourworldindata.org/us-crime-rates
- edoceo 2mo ago
  Didn't freakenomocs claim it (crime rate) was (primarilary) about abortion in the 1970s and less unwanted (and unsupervised/undisciplined) children growing up?
  - deleted 2mo ago
    [deleted]
  - Grombobulous 2mo ago
    There’s also the leaded gasoline theory.
- Grombobulous 2mo ago
  These charts make the exact point I was making. Most of the crime decrease from its peak happened before the year 2000. That means back in the days of law enforcement needing to do low-tech wiretapping techniques, crime was still rapidly decreasing. There’s just not even a correlation. And, as a reminder, correlation is not causation even if it was there.

bottlepalm 2mo ago

Man I thought from the title this was going to be about next-gen AI being able to zero day everything so effectively that software security is meaningless and we'd need to basically shut it all down, go dark.

wseqyrku 2mo ago
[flagged]
- willturman 2mo ago
  Won’t someone please think of those poor helpless law enforcement agencies! AI code is flawless and impenetrable!
  - wizzwizz4 2mo ago
    Well, it's certainly impenetrable.

jrflowers 2mo ago

> I’m concerned that AI is going to make software much too secure. Lmao this is like “I’m concerned the raccoons that I see in the storm drains are going to make our sewer system much too efficient”

cadamsdotcom 2mo ago

We will know we've made systems secure when laws focus on compelling people to provide access. These laws exist - they aren't the focus yet. Right now there's still no need; just hack the device or compel the cloud service to give the data, why waste energy getting consent from its owner! More bugfinding AI, more end to end encryption, more CVEs and more fixes, cannot happen soon enough.

dangoodmanUT 2mo ago

> In April, Anthropic announced a new model called Mythos that was optimized for software vulnerability finding No, it was just good at it because it wasn't RL'd against it. I know this is a small detail, but it tosses journalistic credibility in my eyes.

lazyasciiart 2mo ago
RL'd?
- Ancapistani 2mo ago
  Reinforcement learning, I'd assume. Presumably RLHF (Reinforcement Learning from Human Feedback).
embedding-shape 2mo ago
Even after the initial leaks, Anthropic themselves say they've improved on cybersecurity amount other things, giving the perspective (even if not 100% clear) that one of the focuses was vulnerabilities: > In response to questions about the draft blog post, the company acknowledged training and testing a new model. “We’re developing a general purpose model with meaningful advances in reasoning, coding, and cybersecurity,” an Anthropic spokesperson said. - https://fortune.com/2026/03/26/anthropic-says-testing-mythos-powerful-new-ai-model-after-data-leak-reveals-its-existence-step-change-in-capabilities/ https://fortune.com/2026/03/26/anthropic-says-testing-mythos... It is possible it is both, they used to RL against cybersecurity, but also didn't explicitly do any qualitative tests and added/changed more data because of those results. For Mythos, they stopped RL'ing against it, and also now intentionally try to make it better. Unless of course they've actually noted exactly how things were trained here in some technical report and I've missed it, that's possible. Anthropic aren't famous for being very public about their internals though, but would be curious to read more details about it if it's out there from the horse's mouth.
tptacek 2mo ago
Matthew Green isn't a journalist, he's a practitioner (and a cryptography professor at Hopkins).

hn_submit 2mo ago

I predicted a long time ago that if computers become unhackable LEA and intelligence agencies will push for laws that require backdoors to be built into hard- and software. It will be interesting to see if my prophecy becomes reality. BTW I also hate that Hacker News is being dominated by articles on A.I. lately. Maybe we should vote on HN reducing or even eliminating A.I. related news?

teravor 2mo ago

I don't think the thesis that a government will be able to do something will ultimately hold. I don't see how they can avoid "going dark" in a democracy. we live in a world where the government can't even do much about illegal drug markets anyone can access by downloading a piece of software. if they pass laws that mandate backdoor access and block software which doesn't conform more and more people will move to the dark networks. and if they effectively block the dark networks (in the limit they will have to block all encrypted communications) then we will be living in a tyranny. freedom is messy. accept that digital crime can only be solved when the criminal makes a tangible mistake. LLM's will be building profiles on criminals to help with identifying mistakes.

tptacek 2mo ago
The problem that Matthew Green is calling out subtextually is that democracy is likely to disappoint nerds on this issue: a random voter in the country 10 years from now may very well not share your priors about this issue.
- teravor 2mo ago
  consider the implications of a government where setting up an encrypted network is a serious offense, or where detecting encrypted communications is routine. the average voter doesn't matter. if they are willing to go that far it's a tyranny and the average voter doesn't matter anyway because they spy on everyone and have LLM's profile everyone (in that context even if voting still happens it doesn't matter, they will nudge the voters however they wish due to the information asymmetry).
  - DangitBobby 2mo ago
    It doesn't need to be a serious offense. The current playbook works fine. Anything the Constitution prevents you from doing, have a private company do it for you. In this case you lean on, pay, or give preference to private companies to install back doors. Have the SC rule that you can't sell devices with certain encryption guarantees anywhere on the planet because planetwide commerce affects interstate commerce. Things like that.

ixxie 2mo ago

Some people point out AI can cause bugs as well as fix them, and its a valid point. But the question is: what will the ratio be? If automated pentesting in PR review CI pipeline will become table stakes - which is very plausible - maybe the OP has a point.

Animats 2mo ago

It's worth realizing that, before computerized central offices, telephone wiretapping required running physical wires. Back when Rudi Giuliani was prosecuting organized time, not only did physical wires have to be run, the cops were billed for them as expensive private lines. His task force was spending about a million dollars a year with New York Telephone on wiretapping. In one case, law enforcement didn't pay their bill, resulting in the person being wiretapped having the wiretap connection show up on their bill, blowing the case. That resulted in the Communications Assistance to Law Enforcement Act, which mandated that central offices offer remote wiretapping. Capacity up to 1% of lines is required. Back in the electromechanical era, the only call data that could be collected was outgoing dial pulses, using a "pen register".[1] (The one shown in Wikipedia is mine. It's a beautiful piece of antique brass telegraph technology. It records dial pulses as dashes, and has to be wound up like a clock, with a big brass key.) The Supreme Court decision allowing "pen registers" without a warrant refers to these "extremely limited" devices. That definition has been stretched and stretched by law enforcement into all non-voice data collected by telcos. Law enforcement still wants more. [1] https://en.wikipedia.org/wiki/Pen_register https://en.wikipedia.org/wiki/Pen_register

downrightmike 2mo ago
"Back when Rudi Giuliani was prosecuting organized time" I can't even fathom he'd not be cashing checks from the mafia to go after other families. Odds are good that historians will find that he did.
- ericjmorey 2mo ago
  He traded Italian organized crime for Russian organized crime.
  - usumgallu 2mo ago
    [dead]
- arjie 2mo ago
  A similar story around this is the controversy around various Mumbai “encounter specialists” (a violent incident between police and gangsters is an “encounter”) and whether they were effectively hitmen for some faction of the local mob. Always thought it was quite intriguing stuff.

TechDebtDevin 2mo ago

[dead]

Lerc 2mo ago

Considering from a point of view stipulating that perfectly secure software is possible. I don't think it follows that the limititation that it would place on intelligence is necessarily a net loss. Apart from the obvious harms of invasion of privacy, and fishing expeditions being biased to the places you decided to fish. There is the simple fact that data can be misleading, especially without context. An interceped communication is a piece of data that is intrinsically tied to the trust of the inteceptor. A few people with an agenda can collaborate to create a seeming truth by 'discovering' the same thing from different sources. Requiring warrants compelling information holders to provide data, not only serves the task of protection from abuse but also create a record of provenance that can be verified. It also provides a degree of symmetry in capabilities which discourages actions that one party may do over another if they are motivated to act because they have a temporary advantage over another.

trhway 2mo ago

article read to me like a typical rehash of a typical offense-defense cycle. AI would have written a better article. I personally welcome such spiraling offense-defense cycles as it is one of the main drivers of the technological progress.

ayaros 2mo ago

Governments and their agencies shouldn't have any access to the communications of private citizens. If something like terrorism is the issue, the correct response is to use the wealth and resources of the government to address the root causes of those attacks, not to undermine my right to privacy.

bigDinosaur 2mo ago
This is untrue when private citizens are in conflict (legal or physical or whatever) and the dispute needs to be mediated or resolved. The 'root cause' may well be she said/he said and the legal system may well have need to resolve this by looking at private communications. Strong emphasis on the legal system part, but you've completely dismissed that private communications frequently are required to address causes of issues and the word subpoena makes it rather clear that refusing to comply comes with penalty.
- Chris2048 2mo ago
  In that case, government still doesn't have access; they have to be granted it, even if there is a penalty for non-compliance.

maxo133 2mo ago

I completely disaggree with this take. Modern AI is not yet capable of finding multi-component bugs as advanced as those produced by firms like NSO.

pineapplepizza6 2mo ago
Yes it is, it hacked Artifactory to get to Hugging Face.
- maxo133 2mo ago
  They are not. Developing modern 0-day zero click RCE exploits chains like those developed for iOS is far more complex than hacking to generic company servers It's completely different scale of difficulty factor. Not a single documentated case of of any AI tool developing such exploit exists There is a reason why those mobile 0-day exploits are sold and bought on exploit gray market for as much as 10-20 million dollars each
  - inigyou 2mo ago
    Why?

deleted 2mo ago

[deleted]

bloaf 2mo ago

I would like to point out that the last year when not a single law enforcement agency anywhere in the world could have possibly tapped anyone's phone was 1876. 150 years ago was the invention of the telephone, and I think that articles like this seem to assume that prior to this, police just never caught any criminals.

jMyles 2mo ago
> 1876 A year when police were relatively rare, and quite new in the western legal experience, having only emerged from slave patrols in the United States and from the founding of the 1829 Scotland Yard in the United Kingdom. This is a year with living memory of a time when the state did not employ people to do what was ostensibly the civic responsibility of every person to quell crime and protect others.
- sam345 2mo ago
  So you are advocating no police? You want people to police themselves? .Good luck with that.

RajT88 2mo ago

It breaks my heart that the governments with unlimited budgets who have hired the best and brightest will have to put in serious effort to get the bad guys, and potentially find it not worth it to casually spy on the whole world. I am just beside myself at such an idea that people looking to feed the prison machine cannot as easily find excuses to turn normal citizens into prison feed. Just super sad guys.

xp84 2mo ago
> excuses to turn normal citizens into prison feed What felonies are those normal citizens being convicted of that's landed them in prison? I get the basic 'not wanting everyone's comms to be monitored' part. I think there's an important conversation to be had because indeed, that can't be the answer. But it sounds like you're saying most people who are convicted of felonies based on a jury unanimously being convinced by the electronic evidence against them... those people are unjustly in prison?
- inigyou 2mo ago
  The one who threw a ham sandwich at a Nazi at a protest?
  - GaryBluto 2mo ago
    You mean the guy who assaulted a federal officer and got off scot-free? Woe is him.
    - Nextgrid 2mo ago
      That's selective enforcement though. It's only "assault" because it was done against a cop. It's "just being a dick" when done against anyone else. The danger of constant surveillance is that a lot more of these "being a dick" incidents will be prosecuted when politically convenient (because equal, fair prosecution of all incidents would not happen as that would quickly cause the law to be changed, preventing its future use selective enforcement).

mitchell5584dm 2mo ago

[flagged]

mitchell5584dm 2mo ago

[flagged]

0xDEFACED 2mo ago

i wonder how many open source projects have alphabet boys building trust as contributors for eventual backdoor planting

fitblipper 2mo ago

I've always loved the ridiculousness of the "going dark" label when law enforcement can't access encrypted chats or a back door isn't built into a piece of software. When there are security cameras on the vast majority of houses, stop lights and in people's hands, and when so much meta data about people's associations are shared from Google, Facebook, any other social platform, how in the world can they say they are "going dark". How did they ever solve crimes before these things?

jrowen 2mo ago
Yes, are there any data points to suggest that this arms race is balanced any differently than it ever has been throughout history? The cops will find a way, the criminals will find a way. It's ultimately people v. people with access to the same level of technology.
- graemep 2mo ago
  and everyone else will fall further and further behind
sam345 2mo ago
Wire taps That's the point.

noisebuffer 2mo ago

Without vulnerability derived backdoors I am convinced the government will strong arm the corporations to build a backdoor for the three letter agencies—at least in the US. But we also are at the dawn of quantum systems which could make interception and spying impossible or at least made obvious to the user when it happens. Exhausting infrastructure vulnerabilities even without quantum could be a game changer for many technologies and enable things we can’t do right now, like vote on our phones.

parapsychic 2mo ago
But isn't that the point the writer is making? They'll be self-sabotaging themselves once they do that.
- noisebuffer 2mo ago
  I don’t exactly see how, if there’s no exploit. Either way there’s other realms it’s relevant like armed drones policing a street. Without a way for the public to activate a shutdown sequence, it becomes very difficult to stop if it starts murdering innocent children on some glitch. Which you cannot have because then the whole purpose is ineffective. Likewise digital communication is purposeless and face competing mediums for sure
  - parapsychic 2mo ago
    hmm fair. so a foolproof backdoor that cannot be exploited by the enemies. I don't know how much of that is possible, but then again, the author is talking about super-secure-no-exploit software. So by that definition, a super secure exploit-free backdoor is also possible.

twothreeone 2mo ago

But Matthew, think of the kids!!1 Honestly though, framing this as a "tech issue" doesn't help IMHO, it just muddies the water. Ever since RSA was invented privacy has been about educating people on how to use it effectively and _why they should care_. If voters now are choosing authoritarianism over democracy and individual freedom, I think we have to face the reality that after almost 50 years of fighting battle after battle on the technology front, we've largely lost the war on the home front in this regard.

newHempter 2mo ago

[flagged]

newsomix9xl 2mo ago

So I've read dozens of complaints that AI produces insecure code. I, for one, usually tell my AI to start with secure code, make it small, and modular. This is the first article I've seen that now says the opposite ! AI will make code too secure! Since the small amount of AI coding I've done often results in buggy code (even a shell script written today) with the AI go-to solution of "write more buggy code to fix", this seems counterintuitive.

elisbce 2mo ago

Considering the amount of AI slop being generated today and LOCs that no one actually reviewed, I doubt we will run out of vulnerabilities to exploit...

firefax 2mo ago

Nononono no more of this shit I remember walking into some shitty congresscritter's office with a fucking years old one pager, with a few more citations written on the bottom in pen because I wasn't going to bother making it pretty this time around. You should have seen his face when i asked him straight up: dude, you seem to have a problem processing information. Are you having some kind of medical issue? Because I'm not the last staffer: If you abuse my time, I am never coming back here again to add more citations to a fucking one pager from 1999 -- I'm making it my mission to remove you if you fuck this up on purpose ever again. (Or something to that effect -- I've been told I can get a bit aggressive in my rhetoric.) This was approximately 2016 and that individual is no longer in office. I stand by my words.

rowyourboat 2mo ago
What one pager, what words? This story needs more context if it is going to make sense
- firefax 2mo ago
  No, I think the beauty of it is the ambiguity, but I think there's a culture here that is distinct and different from the creative nonfiction community, so I'll try to be more mindful of whether the site is blue or orange.

fenestella 2mo ago

[flagged]

pelasaco 2mo ago

> So what do we do about it? I just hope that politicians don't jump in and say "we know how to fix it", because unfortunately where they put their hands, they just make things worse (regardless of which country)

getcrunk 2mo ago

I think the authors treatment of backdoors and exceptional access is a narrow focus. We will certainly see “front door” access as various places have or are attempting to enact client side scanning, id verified online access, software root of trust remote attestation, and the general attack on e2e. Taken together your identity tied to unmodifiable software, necessary to access the web or modern economy with client side scanning and no e2e … is a front door. All this has nothing to do with ai, bugs, hackers or security vulnerabilities, let alone backdoors

password4321 2mo ago

For the HN 14 day record since I didn't see this yet: in genuinely critical situations, if the threat model includes a device/service being used against one's self, "the only winning move is not to play" (stop the use of the device/service!) In the same manner, if the critical concern is a device/service being used against one's self remotely, stop the use of the device/service if it is capable of remote access or data collection for later retrieval. The ultimate baseline reality is that deciding to care whether or not remote access/data retrieval is using a method supported by the device manufacturer/service provider and/or whether doing so is allowed by the current legal system is too late. Premptive HN disclaimers: I acknowledge the growing unavoidability of devices and services supporting remote access and/or data collection as I type this here on my phone. Remote access and data collection are very convenient and somehow still expected by device owners and service users to be under their control alone. It can be difficult to determine if a device or service supports remote access/data collection. There are often immediate consequences and eventual legal consequences for stopping the use of devices or services that belong to others. Working to change the legal system to slow/reduce use of devices and services against one's self and encouraging others to do the same is admirable.

password4321 2mo ago

Doesn't this just mean criminal suspects will be forced to unlock their devices using biometrics (in the US)? That should buy law enforcement some more time until biometrics go out of favor.

coldtea 2mo ago

This guy is worried that we might stop having exploits, and that's somehow ...a bad thing? I thought I've read the worst takes about most things, but this still deserves some kind of prize.

caiquelira 2mo ago
I think the problem is that, without organic exploits, we will have to create backdoors on USA systems, making our systems especially vulnerable in the global ecosystem.
- coldtea 2mo ago
  How about not creating/forcing ANY exploits?

happosai 2mo ago

I think US law enforcement will just continue to wiretap everyone at the "SSL added and removed here" SaaS known as cloudflare.

gz5 2mo ago

tcp/ip itself is the ultimate backdoor, similar to the pstn backdoor in the post. genai doesn't change that anytime soon?

Eleg007 2mo ago

Best-written article I've read all week.

trinsic2 2mo ago

> The destruction of the low-hanging vulnerability fruit will make law enforcement (and intelligence) agencies’ need much more acute. The demand for constructed, intentional backdoors will re-start in earnest. Maybe I am overlooking what the author intended but I feel this has already happened and may be the reason why there is so much pressure to create the surveillance state. I feel more like the push for AI to provide better security (everything?) is an excuse to create a paradigm shift with the ultimate purpose of replacing free societies with fiefdoms of corporate culture returning us to the dark ages but with technology running the show. It started way before AI. There are good uses of the technology, but obviously we have a culture of criminal accelerationists that are going to use this overpowered tech to shape society into a system of slavery.

tipsytoad 2mo ago

I think this comes in with the wrong assumptions from the start. The thing that US vs Apple taught us is to not demand access publicly, this puts companies in an awkward spot. If approached more tacitly, gag order etc the company has nothing to gain but everything to lose.. and more likely to comply. I hugely doubt that intentional backdoors don’t exist for the most powerful countries / people

tomrod 2mo ago

I'm going to be a little epistemically silly here. Why doesn't Goedel's incompleteness theorem apply to the assertion that AI-enabled patching will prevent all exploits? I'm quite in favor of impenetrability of software as the alternative is constant friction, to be honest.

franga2000 2mo ago

I don't get it. They can get access to all of our shit through vulnerabilities, but nobody knows about it. In the author's hypothetical world, they get access to all our shit through backdoors, which they have to introduce through a public legislative process, so everyone knows about it. If they can get access to all our shit, at least make them say it out loud and put in the effort to get legislation passed. There's no upside to the current situation. And it's not like they're not trying to introduce backdoors already... Seen the whole age verification thing?

simonebrunozzi 2mo ago

I strongly disagree with this view. The US government will most certainly convince OpenAI, Anthropic and Google to make sure that their models leave a backdoor in most systems that they code. I don't know how, but that's a problem that a team of people paid tens of millions a year each will try to solve. There is NO WAY the US government will let most of the software world go dark.

senordevnyc 2mo ago
I’m guessing you didn’t read to the end, because that’s exactly what the author argues.

the_wolo 2mo ago

As an anti imperialist, I can't wait for US secret services loosing their capabilities :) by the national security that the tech oligarchs (and this article, too) are constantly talking about, they just mean their own security anyway. It's not for you and me, duh.

MetroWind 2mo ago

Shrug~ I'm sure they'll find a way. They have power; they don't need more.