3 ms·

Yes your Chinese open model could have a time-release backdoor, just as your Chinese vibrator could have a hidden microphone that records everything you say and

Yes your Chinese open model could have a time-release backdoor, just as your Chinese vibrator could have a hidden microphone that records everything you say and transmits it to the CCP. But does it? No. What's much more likely is that your US AI provider is promising not to train on your data but is doing so anyway. With a self-hosted model you can at least avoid that.

Barbing 2mo ago
Always assume someone’s training on ya (maybe not to point of losing competitive advantage, but on a personal level). Airgap your models from Country A, review outputs with model from Country B?
- nerdsniper 2mo ago
  They’re talking about open-weight models you host yourself. Which are fundamentally incapable of spying on you unless you give them network access. We also haven’t seen any evidence of open-weight models exfiltrating data over side channels (DNS/steganography/etc) yet, so at this time internet access still seems safe enough to grant without worrying about spying.
  - dvh 2mo ago
    I can imagine scenario. Let's say you tell open-weight model to generate kitten video. It uses piezoelectric effect of some ceramic capacitor on gpu as microphone and adds recorded audio to the generated video in non audible range. You post kitten video on social media where ai company collects it and extract what you were saying while the video was being generated.
    - cyanydeez 2mo ago
      I too believe in paranoid magic.
  - Barbing 2mo ago
    Degraded/sabotaged responses also present some risk. Some scenarios mentioned here: https://news.ycombinator.com/item?id=49416695 https://news.ycombinator.com/item?id=49416695
Icy-Undivided-G 2mo ago
my first thought reading the article wasn't that Qwen, or some other chinese open model, would do this, but rather that one of the many "unlocked" models on huggingface might
- embedding-shape 2mo ago
  But but but, they're called .safetensors, surely they're safe then and could impossibly lead to me being hacked? The ecosystem is due for a rude awakening any day now. I've been slowly prepping by isolating absolutely everything on my machine from each other, not sure how people dare to YOLO run these sort of things "natively" on their computer today.
dannyw 2mo ago
> What's much more likely is that your US AI provider is promising not to train on your data but is doing so anyway. There's genuinely no evidence of this for OpenAI and Anthropic. It's impossible to disprove, but I don't think it's likely because: * They get enough volume from consumer subs with data training enabled anyway. * If this was happening, it needs serious work at the scale OpenAI and Anthroppic, from data pipelines, to ablation experiments, to the actual data mix and traces going in all the telemetry/diagnosis of large-scale training runs. * It would need to involve a team. Employees at these companies leave, there have been numerous whistleblowers, allegations, etc. Nothing on this front that I can find. * It would damage enterprise trust permanently and be a company and reputation-ending thing. Now that these tools are used by everyone from state governments to the DoW, the exposure radius is massive, investors (many of whom are customers/users too; and often have their stakes in not just a single company but multiple) would not be happy. Piss off enough powerful people, and anyone can join Sam Bankman-Fried in prison. * There's a myriad of enterprise customers and bespoke contracts. I can't get into details, but not all enterprises accept a 'trust me bro' clause.
- jjav 2mo ago
  > It's impossible to disprove, but I don't think it's likely If something is impossible to disprove, we must assume it is happening from a threat modeling perspective.
- zarzavat 2mo ago
  I'm not saying that Anthropic and OpenAI are definitely dishonest, there is no proof of that. However, when you have 1) an extremely high financial incentive to be dishonest (trillions of dollars), 2) low-ish chance of being caught, especially if you launder the data through another model to remove identifying information, 3) the people in charge of said operations are generally agreed to be snakes, then it should at least arouse suspicion. You may be right that get enough data from opt-ins that they don't need to do it. I do believe that they don't violate enterprise ZDR agreements, but for normal subscriptions I'm much less confident.
- dgellow 2mo ago
  > * It would need to involve a team. Employees at these companies leave, there have been numerous whistleblowers, allegations, etc. Nothing on this front that I can find. You have to also note the incentives going on here. Employees who have extremely valuable stock options, they have good reasons to not come public with anything that would damage the reputation of the company before they can cash out
skeledrew 2mo ago
> your Chinese vibrator could have a hidden microphone that records everything you say and transmits it to the CCP Could they be that desperate? Wow.
- kkfx 2mo ago
  https://www.theguardian.com/technology/2017/mar/14/we-vibe-vibrator-tracking-users-sexual-habits https://www.theguardian.com/technology/2017/mar/14/we-vibe-v...
  - follower 2mo ago
    Although, for completeness, the company in that case is actually Canadian. (Always nice to see decade old effort still get referenced on occasion. :) ) Source: Username checks out. :D
    - jimz 2mo ago
      And less than a decade later we have multiple devices that are effectively fitbits for your genitalia, for both sexes. How times change.
cyanydeez 2mo ago
Your US AI provider may be actively doing what AT&T started doing after 911, but you know, under the permission structure of fascism.

Barbing 2mo ago

Always assume someone’s training on ya (maybe not to point of losing competitive advantage, but on a personal level). Airgap your models from Country A, review outputs with model from Country B?

nerdsniper 2mo ago
They’re talking about open-weight models you host yourself. Which are fundamentally incapable of spying on you unless you give them network access. We also haven’t seen any evidence of open-weight models exfiltrating data over side channels (DNS/steganography/etc) yet, so at this time internet access still seems safe enough to grant without worrying about spying.
- dvh 2mo ago
  I can imagine scenario. Let's say you tell open-weight model to generate kitten video. It uses piezoelectric effect of some ceramic capacitor on gpu as microphone and adds recorded audio to the generated video in non audible range. You post kitten video on social media where ai company collects it and extract what you were saying while the video was being generated.
  - cyanydeez 2mo ago
    I too believe in paranoid magic.
- Barbing 2mo ago
  Degraded/sabotaged responses also present some risk. Some scenarios mentioned here: https://news.ycombinator.com/item?id=49416695 https://news.ycombinator.com/item?id=49416695

Icy-Undivided-G 2mo ago

my first thought reading the article wasn't that Qwen, or some other chinese open model, would do this, but rather that one of the many "unlocked" models on huggingface might

embedding-shape 2mo ago
But but but, they're called .safetensors, surely they're safe then and could impossibly lead to me being hacked? The ecosystem is due for a rude awakening any day now. I've been slowly prepping by isolating absolutely everything on my machine from each other, not sure how people dare to YOLO run these sort of things "natively" on their computer today.

dannyw 2mo ago

> What's much more likely is that your US AI provider is promising not to train on your data but is doing so anyway. There's genuinely no evidence of this for OpenAI and Anthropic. It's impossible to disprove, but I don't think it's likely because: * They get enough volume from consumer subs with data training enabled anyway. * If this was happening, it needs serious work at the scale OpenAI and Anthroppic, from data pipelines, to ablation experiments, to the actual data mix and traces going in all the telemetry/diagnosis of large-scale training runs. * It would need to involve a team. Employees at these companies leave, there have been numerous whistleblowers, allegations, etc. Nothing on this front that I can find. * It would damage enterprise trust permanently and be a company and reputation-ending thing. Now that these tools are used by everyone from state governments to the DoW, the exposure radius is massive, investors (many of whom are customers/users too; and often have their stakes in not just a single company but multiple) would not be happy. Piss off enough powerful people, and anyone can join Sam Bankman-Fried in prison. * There's a myriad of enterprise customers and bespoke contracts. I can't get into details, but not all enterprises accept a 'trust me bro' clause.

jjav 2mo ago
> It's impossible to disprove, but I don't think it's likely If something is impossible to disprove, we must assume it is happening from a threat modeling perspective.
zarzavat 2mo ago
I'm not saying that Anthropic and OpenAI are definitely dishonest, there is no proof of that. However, when you have 1) an extremely high financial incentive to be dishonest (trillions of dollars), 2) low-ish chance of being caught, especially if you launder the data through another model to remove identifying information, 3) the people in charge of said operations are generally agreed to be snakes, then it should at least arouse suspicion. You may be right that get enough data from opt-ins that they don't need to do it. I do believe that they don't violate enterprise ZDR agreements, but for normal subscriptions I'm much less confident.
dgellow 2mo ago
> * It would need to involve a team. Employees at these companies leave, there have been numerous whistleblowers, allegations, etc. Nothing on this front that I can find. You have to also note the incentives going on here. Employees who have extremely valuable stock options, they have good reasons to not come public with anything that would damage the reputation of the company before they can cash out

skeledrew 2mo ago

> your Chinese vibrator could have a hidden microphone that records everything you say and transmits it to the CCP Could they be that desperate? Wow.

kkfx 2mo ago
https://www.theguardian.com/technology/2017/mar/14/we-vibe-vibrator-tracking-users-sexual-habits https://www.theguardian.com/technology/2017/mar/14/we-vibe-v...
- follower 2mo ago
  Although, for completeness, the company in that case is actually Canadian. (Always nice to see decade old effort still get referenced on occasion. :) ) Source: Username checks out. :D
  - jimz 2mo ago
    And less than a decade later we have multiple devices that are effectively fitbits for your genitalia, for both sexes. How times change.

cyanydeez 2mo ago

Your US AI provider may be actively doing what AT&T started doing after 911, but you know, under the permission structure of fascism.