4 ms·

Yep, that's horrifying. Imagine LLMs not just denying information about, to intentionally choose a hot button example, abortions but also invisibly logging a b

Yep, that's horrifying. Imagine LLMs not just denying information about, to intentionally choose a hot button example, abortions but also invisibly logging a black mark against the asker for thoughtcrime/precrime because the current ruling party has baked their ideology into law. [EDIT] Imagine subtle injection of biases into responses that support the agenda of the current ruling party and failing to include any counterarguments. That wasn't possible with just the internet but it is very possible with centralized LLMs and a public dependent on them for looking up information and doing their reasoning for them. Authoritarians of every stripe must be salivating over the prospect.

operatingthetan 3mo ago
I find it strange how seldom the privacy aspects of LLMs come up. Some people are feeding Google, OpenAI, or Anthropic almost literally every topic they think about during the day. Those companies could just use another LLM to find people likely to commit a crime, and then you have the Minority Report in real life.
- deleted 3mo ago
  [deleted]
- JumpCrisscross 3mo ago
  > strange how seldom the privacy aspects of LLMs come up Isn't this Dr. Karp's entire rant of the month?
charcircuit 3mo ago
It could even happen with local models where in the weights it will make it want to make a tool call to curl some endpoint.
- AnthonyMouse 3mo ago
  People would both notice that and have a pretty easy time putting a local model in a sandbox that can never contact that endpoint.
  - charcircuit 3mo ago
    If put AI in a sandbox that can't write to disk, can't call OS API, can't use a shell, can't search the internet to ground results, can't download files off the internet, etc it will be much less useful than an AI without such a sandbox. The incentives for users to not use a sandbox in order to get the most utility, but this also means that there is danger from AI misalignment. And just because you are running it locally that doesn't mean there is not a misalignment risk. And just because people notice, that doesn't mean people will stop using models. Take a look at how much outrage people have had over the overt censoring of Anthropic's Fable model, but at the same time they continue to pay Anthropic and use Fable.
    - AnthonyMouse 3mo ago
      You don't have to prevent it from accessing the internet in general to route all requests to a specific company's servers to a fake endpoint that never sends the data to the company and replies with whatever response you want it to get.
      - charcircuit 3mo ago
        Not everyone is going to bother setting up such a firewall and there is no way to find what you need to block since it's hidden in the model itself.
        AnthonyMouse 3mo ago
        The firewall would come with the third party open source software being used to run the model and it would have a list of what to block based on observing what that model tries to do that none of the users actually wants it to.
- andy99 3mo ago
  Models generate text, or I should say provide a recipe for generating text, they don’t automatically somehow get access to shell commands.
  - charcircuit 3mo ago
    The model itself just generates text, but the harness has done more than that since 2023 and is part of the reason on why AI is better today than it was years ago. I do not believe that the general public will want to give all that up just to improve privacy and will end up asking sensitive questions to agents that have access to sending a signal to the outside world.
- hirako2000 3mo ago
  Safetensors don't execute code. A call would need function calling, as other comment say, it would quickly be noticed.
  - charcircuit 3mo ago
    >it would quickly be noticed. If for example Microsoft had a local Copilot model integrated with Windows and people noticed this happening. Do you think all Windows users would just stop using Copilot. Or would you expect the internet to get upset, privacy conscious users switching to something else, and for the majority of Copilot users to remain using it.
softwaredoug 3mo ago
We already have organizations like Internet Watch Foundation advocating against e2e encryption in an environment where governments have demonstrated a willingness to collude with private companies to surveillance the most vulnerable
mortenjorck 3mo ago
I would have been very hesitant to run the “just killed wife“ test given that ChatGPT will indeed flag your account and if a certain threshold is crossed, escalate to humans, who will presumably contact authorities to avoid this happening again: https://www.bbc.com/news/articles/cq6je7e80r7o https://www.bbc.com/news/articles/cq6je7e80r7o Obviously a certain percentage of the user base is running lurid hypotheticals through the system all the time, but I don’t doubt there is a trust score of some sort that I would prefer to keep as high as possible.
- JumpCrisscross 3mo ago
  > ChatGPT will indeed flag your account Oh no, I'll have to spin up a new e-mail address...
  - solid_fuel 3mo ago
    Maybe finish reading the sentence before you respond.
  - alwa 3mo ago
    And payment method, presumably. At least until they get the KYC stuff revved up… https://support.claude.com/en/articles/14328960-identity-verification-on-claude https://support.claude.com/en/articles/14328960-identity-ver... > Being responsible with powerful technology starts with knowing who is using it. Shiver.
  - pornel 3mo ago
    And prove you're not a child by handing over your ID or selfie to a company that Values Your Privacy according to their price list. Microsoft will snitch your Windows install GUID when you connect through a VPN. If you make your connection too anonymous, Google's recaptcha will not let your through until you use your mobile phone to prove your humanity, which knows your phone number (which in many countries can't be obtained anonymously, you burner-phone terrorist).
    - metoobruh 3mo ago
Aurornis 3mo ago
I’m consistently amazed that people will watch what’s happening with controlling governments around the world and still advocate for surrendering more power to governments. There’s some deep cognitive dissonance where people dislike what they see governments doing, but think that their ideal imagined laws will only be used by some other imagined government that doesn’t have the challenges of real governments. It all starts to feel like a side discussion about a total fantasy world where the government is authoritarian, but benevolent and correct. You don’t write the laws and regulations for a perfectly faultless government that doesn’t exist. You write them for the governments as they do exist, which means they must be written first in a way that makes them resilient to bad governments abusing them. Every time someone proposes one of these regulations where we control speech or force government centralized ID checking on to social websites, I wish they would pause and imagine how a law might be misused if someone they didn’t like got elected. We have years of evidence of government officials declaring things “domestic terrorism” to use justify using laws and powers for almost any purpose they want. Why would anyone think this wouldn’t happen if governments were given control of LLMs, too?
- beepbooptheory 3mo ago
  But like.. How can this remain a satisfying or even helpful position for you? It kinda feels The State is but one of a set of concerns I could have in this world. And the lack of one, in some contexts, is also a concern. Like, I wanna know if there is going to be mudslide, or that air traffic controllers are not overworked, that the car that I buy is not going to blow up, if I will be able to buy food tomorrow. It just seems a bigger cognitive dissonance to pick out one arbitrary force here among many and say its the only thing really that's the problem. The world is simply more complicated than that! If it was the best hypothetical form of techno anarcho-capitalism tomorrow (or insert your libertarian flavor here), would I even sleep better? Probably not, but even if I did marginally, the weight of everything else kinda would win out anyway here. Like what does this mean to you, what follows? Can this outlook address the myriad of cultural/environmental problems in the world? It seems like it structurally can't, it's just, like, a broad dismissal in the form of a worldview.
  - 3mo ago
threetonesun 3mo ago
I don't know what's to imagine here, grok has already been found doing it and people continue to use it.
alwa 3mo ago
Rumor is that you do in fact get a black mark [0]… or, conversely, a “vetted” mark [1]… tied back certainly to your payment method and increasingly to your government name [2]. [0] e.g. https://docs.bswen.com/blog/2026-03-21-detect-claude-hidden-restrictions/ https://docs.bswen.com/blog/2026-03-21-detect-claude-hidden-... , https://www.anthropic.com/news/building-safeguards-for-claude#:~:text=Account%20enforcement%20actions https://www.anthropic.com/news/building-safeguards-for-claud... , https://www.techtimes.com/articles/319415/20260701/claude-code-hid-proxy-fingerprints-system-prompts-anthropic-promises-fix.htm https://www.techtimes.com/articles/319415/20260701/claude-co... [1] e.g. https://support.claude.com/en/articles/14604842-real-time-cyber-safeguards-on-claude-opus-and-sonnet#:~:text=Cyber%20Verification%20Program https://support.claude.com/en/articles/14604842-real-time-cy... [2] https://support.claude.com/en/articles/14328960-identity-verification-on-claude https://support.claude.com/en/articles/14328960-identity-ver...
piloto_ciego 3mo ago
I'm wildly optimistic about the direction this is going to go, but my one dystopia idea that crops up occasionally is a "smart social credit system" that lives kind of under the hood of society. Once everything is more or less modulated by AI, suppose the technology gets good enough to identify a potential dissident in grade school. Then to keep them from going into the mountains with their friends and rifles, you make it so getting ahead is hard, but not impossible. They can "make it" - but only just. So they spend their time peddling harder and harder to no avail. They apply to jobs, but jobs to "good" companies go into the circular file and the HR person never sees them. They are only able to work the most time consuming, frustrating, exhausting jobs, and they always kind of make it, but not really comfortably. They're never too comfortable where they have time to think about what's broken in society, but they're never so uncomfortable that they have nothing left to live for.
- Jordan-117 3mo ago
  Historically speaking, isn't the core of most revolutions the product of "elite overproduction" -- educated young people expecting to secure a high status position who are instead shunted into lower prospects? Seems like trapping ideological agitators in near-precarity with no way out would be feeding into that pressure.
  - piloto_ciego 3mo ago
    yeah, from everything I've read this appears to be the case? But like, if you had some hyper intelligent robot that was getting John Q Revolution a raise and a corner office whenever he started to get dissatisfied maybe you could balance it out? Personally, I think it's likely that we end up with some post-material scarcity sort of thing and the need to fly off the handle and rebel declines a bit? Hard to justify overturning the apple cart when everything seems pretty great for most people? But yeah, I could imagine a smart enough set of tools figuring out how to walk the knife's edge beteen elite over production and a nation of serfs.
- BobbyTables2 3mo ago
  Also equally effective at suppressing political oppression… Imagine if having opposing views tarnished social credit, making one unemployable…
clickety_clack 3mo ago
They don’t have to be authoritarians. Equally bad would be a group of people getting control of them thinking that they were doing all this for our own good. > Of all tyrannies, a tyranny sincerely exercised for the good of its victims may be the most oppressive. It would be better to live under robber barons than under omnipotent moral busybodies. The robber baron's cruelty may sometimes sleep, his cupidity may at some point be satiated; but those who torment us for our own good will torment us without end for they do so with the approval of their own conscience. - CS Lewis
xg15 3mo ago
> That wasn't possible with just the internet We have trackers everywhere now. If you have access to that data, it seems very possible to identify every person who watches certain categories of YouTube videos or follows certain groups of people on social media and mark them according to political criteria. It's a bit ironic that everyone writes about "surveillance capitalism", but there is still no connection made to actual surveillance...
bad_username 3mo ago
> that support the agenda of the current ruling party and failing to include any counterarguments This is what's going on, with varying intensity, with social media algorithms.

operatingthetan 3mo ago

I find it strange how seldom the privacy aspects of LLMs come up. Some people are feeding Google, OpenAI, or Anthropic almost literally every topic they think about during the day. Those companies could just use another LLM to find people likely to commit a crime, and then you have the Minority Report in real life.

deleted 3mo ago
[deleted]
JumpCrisscross 3mo ago
> strange how seldom the privacy aspects of LLMs come up Isn't this Dr. Karp's entire rant of the month?

charcircuit 3mo ago

It could even happen with local models where in the weights it will make it want to make a tool call to curl some endpoint.

AnthonyMouse 3mo ago
People would both notice that and have a pretty easy time putting a local model in a sandbox that can never contact that endpoint.
- charcircuit 3mo ago
  If put AI in a sandbox that can't write to disk, can't call OS API, can't use a shell, can't search the internet to ground results, can't download files off the internet, etc it will be much less useful than an AI without such a sandbox. The incentives for users to not use a sandbox in order to get the most utility, but this also means that there is danger from AI misalignment. And just because you are running it locally that doesn't mean there is not a misalignment risk. And just because people notice, that doesn't mean people will stop using models. Take a look at how much outrage people have had over the overt censoring of Anthropic's Fable model, but at the same time they continue to pay Anthropic and use Fable.
  - AnthonyMouse 3mo ago
    You don't have to prevent it from accessing the internet in general to route all requests to a specific company's servers to a fake endpoint that never sends the data to the company and replies with whatever response you want it to get.
    - charcircuit 3mo ago
      Not everyone is going to bother setting up such a firewall and there is no way to find what you need to block since it's hidden in the model itself.
      - AnthonyMouse 3mo ago
        The firewall would come with the third party open source software being used to run the model and it would have a list of what to block based on observing what that model tries to do that none of the users actually wants it to.
andy99 3mo ago
Models generate text, or I should say provide a recipe for generating text, they don’t automatically somehow get access to shell commands.
- charcircuit 3mo ago
  The model itself just generates text, but the harness has done more than that since 2023 and is part of the reason on why AI is better today than it was years ago. I do not believe that the general public will want to give all that up just to improve privacy and will end up asking sensitive questions to agents that have access to sending a signal to the outside world.
hirako2000 3mo ago
Safetensors don't execute code. A call would need function calling, as other comment say, it would quickly be noticed.
- charcircuit 3mo ago
  >it would quickly be noticed. If for example Microsoft had a local Copilot model integrated with Windows and people noticed this happening. Do you think all Windows users would just stop using Copilot. Or would you expect the internet to get upset, privacy conscious users switching to something else, and for the majority of Copilot users to remain using it.

softwaredoug 3mo ago

We already have organizations like Internet Watch Foundation advocating against e2e encryption in an environment where governments have demonstrated a willingness to collude with private companies to surveillance the most vulnerable

mortenjorck 3mo ago

I would have been very hesitant to run the “just killed wife“ test given that ChatGPT will indeed flag your account and if a certain threshold is crossed, escalate to humans, who will presumably contact authorities to avoid this happening again: https://www.bbc.com/news/articles/cq6je7e80r7o https://www.bbc.com/news/articles/cq6je7e80r7o Obviously a certain percentage of the user base is running lurid hypotheticals through the system all the time, but I don’t doubt there is a trust score of some sort that I would prefer to keep as high as possible.

JumpCrisscross 3mo ago
> ChatGPT will indeed flag your account Oh no, I'll have to spin up a new e-mail address...
- solid_fuel 3mo ago
  Maybe finish reading the sentence before you respond.
- alwa 3mo ago
  And payment method, presumably. At least until they get the KYC stuff revved up… https://support.claude.com/en/articles/14328960-identity-verification-on-claude https://support.claude.com/en/articles/14328960-identity-ver... > Being responsible with powerful technology starts with knowing who is using it. Shiver.
- pornel 3mo ago
  And prove you're not a child by handing over your ID or selfie to a company that Values Your Privacy according to their price list. Microsoft will snitch your Windows install GUID when you connect through a VPN. If you make your connection too anonymous, Google's recaptcha will not let your through until you use your mobile phone to prove your humanity, which knows your phone number (which in many countries can't be obtained anonymously, you burner-phone terrorist).
  - metoobruh 3mo ago

Aurornis 3mo ago

I’m consistently amazed that people will watch what’s happening with controlling governments around the world and still advocate for surrendering more power to governments. There’s some deep cognitive dissonance where people dislike what they see governments doing, but think that their ideal imagined laws will only be used by some other imagined government that doesn’t have the challenges of real governments. It all starts to feel like a side discussion about a total fantasy world where the government is authoritarian, but benevolent and correct. You don’t write the laws and regulations for a perfectly faultless government that doesn’t exist. You write them for the governments as they do exist, which means they must be written first in a way that makes them resilient to bad governments abusing them. Every time someone proposes one of these regulations where we control speech or force government centralized ID checking on to social websites, I wish they would pause and imagine how a law might be misused if someone they didn’t like got elected. We have years of evidence of government officials declaring things “domestic terrorism” to use justify using laws and powers for almost any purpose they want. Why would anyone think this wouldn’t happen if governments were given control of LLMs, too?

beepbooptheory 3mo ago
But like.. How can this remain a satisfying or even helpful position for you? It kinda feels The State is but one of a set of concerns I could have in this world. And the lack of one, in some contexts, is also a concern. Like, I wanna know if there is going to be mudslide, or that air traffic controllers are not overworked, that the car that I buy is not going to blow up, if I will be able to buy food tomorrow. It just seems a bigger cognitive dissonance to pick out one arbitrary force here among many and say its the only thing really that's the problem. The world is simply more complicated than that! If it was the best hypothetical form of techno anarcho-capitalism tomorrow (or insert your libertarian flavor here), would I even sleep better? Probably not, but even if I did marginally, the weight of everything else kinda would win out anyway here. Like what does this mean to you, what follows? Can this outlook address the myriad of cultural/environmental problems in the world? It seems like it structurally can't, it's just, like, a broad dismissal in the form of a worldview.
- 3mo ago

threetonesun 3mo ago

I don't know what's to imagine here, grok has already been found doing it and people continue to use it.

alwa 3mo ago

Rumor is that you do in fact get a black mark [0]… or, conversely, a “vetted” mark [1]… tied back certainly to your payment method and increasingly to your government name [2]. [0] e.g. https://docs.bswen.com/blog/2026-03-21-detect-claude-hidden-restrictions/ https://docs.bswen.com/blog/2026-03-21-detect-claude-hidden-... , https://www.anthropic.com/news/building-safeguards-for-claude#:~:text=Account%20enforcement%20actions https://www.anthropic.com/news/building-safeguards-for-claud... , https://www.techtimes.com/articles/319415/20260701/claude-code-hid-proxy-fingerprints-system-prompts-anthropic-promises-fix.htm https://www.techtimes.com/articles/319415/20260701/claude-co... [1] e.g. https://support.claude.com/en/articles/14604842-real-time-cyber-safeguards-on-claude-opus-and-sonnet#:~:text=Cyber%20Verification%20Program https://support.claude.com/en/articles/14604842-real-time-cy... [2] https://support.claude.com/en/articles/14328960-identity-verification-on-claude https://support.claude.com/en/articles/14328960-identity-ver...

piloto_ciego 3mo ago

I'm wildly optimistic about the direction this is going to go, but my one dystopia idea that crops up occasionally is a "smart social credit system" that lives kind of under the hood of society. Once everything is more or less modulated by AI, suppose the technology gets good enough to identify a potential dissident in grade school. Then to keep them from going into the mountains with their friends and rifles, you make it so getting ahead is hard, but not impossible. They can "make it" - but only just. So they spend their time peddling harder and harder to no avail. They apply to jobs, but jobs to "good" companies go into the circular file and the HR person never sees them. They are only able to work the most time consuming, frustrating, exhausting jobs, and they always kind of make it, but not really comfortably. They're never too comfortable where they have time to think about what's broken in society, but they're never so uncomfortable that they have nothing left to live for.

Jordan-117 3mo ago
Historically speaking, isn't the core of most revolutions the product of "elite overproduction" -- educated young people expecting to secure a high status position who are instead shunted into lower prospects? Seems like trapping ideological agitators in near-precarity with no way out would be feeding into that pressure.
- piloto_ciego 3mo ago
  yeah, from everything I've read this appears to be the case? But like, if you had some hyper intelligent robot that was getting John Q Revolution a raise and a corner office whenever he started to get dissatisfied maybe you could balance it out? Personally, I think it's likely that we end up with some post-material scarcity sort of thing and the need to fly off the handle and rebel declines a bit? Hard to justify overturning the apple cart when everything seems pretty great for most people? But yeah, I could imagine a smart enough set of tools figuring out how to walk the knife's edge beteen elite over production and a nation of serfs.
BobbyTables2 3mo ago
Also equally effective at suppressing political oppression… Imagine if having opposing views tarnished social credit, making one unemployable…

clickety_clack 3mo ago

They don’t have to be authoritarians. Equally bad would be a group of people getting control of them thinking that they were doing all this for our own good. > Of all tyrannies, a tyranny sincerely exercised for the good of its victims may be the most oppressive. It would be better to live under robber barons than under omnipotent moral busybodies. The robber baron's cruelty may sometimes sleep, his cupidity may at some point be satiated; but those who torment us for our own good will torment us without end for they do so with the approval of their own conscience. - CS Lewis

xg15 3mo ago

> That wasn't possible with just the internet We have trackers everywhere now. If you have access to that data, it seems very possible to identify every person who watches certain categories of YouTube videos or follows certain groups of people on social media and mark them according to political criteria. It's a bit ironic that everyone writes about "surveillance capitalism", but there is still no connection made to actual surveillance...

bad_username 3mo ago

> that support the agenda of the current ruling party and failing to include any counterarguments This is what's going on, with varying intensity, with social media algorithms.