6 ms·

There's a higher-order concern here that I'm paranoid enough to voice: that if used as a coding agent, an AI model affiliated with a country's government might

There's a higher-order concern here that I'm paranoid enough to voice: that if used as a coding agent, an AI model affiliated with a country's government might try to make my software susceptible to attacks by that government's intelligence forces. And note that I'm not singling out China here.

stevehawk 4mo ago
so use the cheap model to do the work and the expensive domestic model to audit?
- SpicyLemonZest 4mo ago
  Or I can just use the domestic model, accepting that I'm paying some premium in order to reduce the complexity of my dependencies and the amount of time I have to spend thinking about supply chain risk. It's the same reason I don't buy things from Alibaba even though many things I buy from Amazon are surely available there for less.
  - throw1234567891 4mo ago
    You use “use the model” as if it was equal to “paid some guys to run inference on their hardware”.
add-sub-mul-div 4mo ago
Giving up our agency to AI has the potential to turn us into NPCs, period. Economically, politically, socially. They've invented a vehicle for inserting any idea they want into our consumption and output.
zozbot234 4mo ago
> that if used as a coding agent, an AI model affiliated with a country's government might try to make my software susceptible to attacks by that government's intelligence forces. Note that if such a trigger were to exist, the behavior has to be completely reproducible by definition, e.g. when put into the right setting with the right input context, the model starts behaving maliciously with at least some well-defined probability. I don't think any such incident has ever been described, it's a purely theoretical concern.
- Avicebron 4mo ago
  I don't think it's a stretch that you can train/align a model to avoid "hatespeech" or other topics deemed $Unacceptable you can align a model to favor a certain ideological viewpoint and have that alignment subtly influence the output. How do most Chinese models handle Tienanmen square or discussions on Han superiority?
  - zozbot234 4mo ago
    > How do most Chinese models handle Tienanmen square or discussions on Han superiority? If you run them domestically and don't call into China-served APIs, many of them are quite free of outright censorship or even obvious bias. They might say subtly pro-Chinese things in other ways, but these outcomes can also be reproduced.
  - margalabargala 4mo ago
    Oh sure, no one said you can't train a model to do this. You certainly can. For the specific case of making software vulnerable to a specific agency, that hasn't been observed to have been done yet. Not because it can't be, but because no one has for now. If it were done, it would be easy(ish) to detect, since it'll be reproducible.
    - falcor84 4mo ago
      My flavor of paranoia is not as overt as maliciously adding an exploit, but that whenever there are multiple reasonable ways of designing a solution, it'd choose an approach that is susceptible to one of the zero-days currently known to that country. I don't see how reproducibility would help you there.
    - sometimelurker 4mo ago
      > easy(ish) to detect 100% on small models, but frontier models (at the level ddeepseekv4pro) can tell when their being tested so it becomes harder to check. you can always finetune them to remove CCP propaganda from them
imjonse 4mo ago
Since that is valid for every model from any country, it's a good idea to review the code the agent creates :)
sometimelurker 4mo ago
you can finetune the ccp propaganda out of them, then your mostly fine. if you want to be more safe you can finetune their public base models to not have ccp propagnada, and then proceed with the rest of the training (costs more tho)
moron4hire 4mo ago
Isn't this only a concern for yolocoding? All the AI-advocates tell me that "good" use of AI should include human review. Of course, they never seem able to explain why the boss that makes you use coding agents to go fast wouldn't be the same boss that pressures you to "just ship it, it's working" and skip review, so I absolutely believe your concern is valid.
beepbooptheory 4mo ago
Almost feels like maybe the best bet is to have humans make the code when its really important.
- throw1234567891 4mo ago
  Because people cannot be manipulated.
Humorist2290 4mo ago
It's more comical than sinister, but I have an example in this vein. I was using Claude to work on a pet project which itself has a "generate with AI" feature. The default model the project uses was Gemini (because it was cheaper and more reliably produces the correct output format). Claude kept changing the default model to Opus when working on entirely unrelated parts, and I kept noticing it because Opus would mangle the output and break the rendered page. It also did this to the .env file in addition to the default.
HarHarVeryFunny 4mo ago
If you're that paranoid, then you shouldn't be using any online services at all, and should not have an internet connection to your PC. Never use a compiler that you have not bootstrapped yourself without the use of any other compiler binary. Even with these precautions you may still be hacked by state-level actors using a whole variety of sophisticated attack vectors. There may be Stuxnet-like software hidden on your hard drive where you cannot see it. If you do not have a TEMPEST hardened compute environment then anything you type on your keyboard or display on your screen may be getting stolen. That said, it would be a fantastic achievement if someone could create a coding model that managed to hide a backdoor in the code it was generating. although surely simpler to hack you in 100 other ways.

stevehawk 4mo ago

so use the cheap model to do the work and the expensive domestic model to audit?

SpicyLemonZest 4mo ago
Or I can just use the domestic model, accepting that I'm paying some premium in order to reduce the complexity of my dependencies and the amount of time I have to spend thinking about supply chain risk. It's the same reason I don't buy things from Alibaba even though many things I buy from Amazon are surely available there for less.
- throw1234567891 4mo ago
  You use “use the model” as if it was equal to “paid some guys to run inference on their hardware”.

add-sub-mul-div 4mo ago

Giving up our agency to AI has the potential to turn us into NPCs, period. Economically, politically, socially. They've invented a vehicle for inserting any idea they want into our consumption and output.

zozbot234 4mo ago

> that if used as a coding agent, an AI model affiliated with a country's government might try to make my software susceptible to attacks by that government's intelligence forces. Note that if such a trigger were to exist, the behavior has to be completely reproducible by definition, e.g. when put into the right setting with the right input context, the model starts behaving maliciously with at least some well-defined probability. I don't think any such incident has ever been described, it's a purely theoretical concern.

Avicebron 4mo ago
I don't think it's a stretch that you can train/align a model to avoid "hatespeech" or other topics deemed $Unacceptable you can align a model to favor a certain ideological viewpoint and have that alignment subtly influence the output. How do most Chinese models handle Tienanmen square or discussions on Han superiority?
- zozbot234 4mo ago
  > How do most Chinese models handle Tienanmen square or discussions on Han superiority? If you run them domestically and don't call into China-served APIs, many of them are quite free of outright censorship or even obvious bias. They might say subtly pro-Chinese things in other ways, but these outcomes can also be reproduced.
- margalabargala 4mo ago
  Oh sure, no one said you can't train a model to do this. You certainly can. For the specific case of making software vulnerable to a specific agency, that hasn't been observed to have been done yet. Not because it can't be, but because no one has for now. If it were done, it would be easy(ish) to detect, since it'll be reproducible.
  - falcor84 4mo ago
    My flavor of paranoia is not as overt as maliciously adding an exploit, but that whenever there are multiple reasonable ways of designing a solution, it'd choose an approach that is susceptible to one of the zero-days currently known to that country. I don't see how reproducibility would help you there.
  - sometimelurker 4mo ago
    > easy(ish) to detect 100% on small models, but frontier models (at the level ddeepseekv4pro) can tell when their being tested so it becomes harder to check. you can always finetune them to remove CCP propaganda from them

imjonse 4mo ago

Since that is valid for every model from any country, it's a good idea to review the code the agent creates :)

sometimelurker 4mo ago

you can finetune the ccp propaganda out of them, then your mostly fine. if you want to be more safe you can finetune their public base models to not have ccp propagnada, and then proceed with the rest of the training (costs more tho)

moron4hire 4mo ago

Isn't this only a concern for yolocoding? All the AI-advocates tell me that "good" use of AI should include human review. Of course, they never seem able to explain why the boss that makes you use coding agents to go fast wouldn't be the same boss that pressures you to "just ship it, it's working" and skip review, so I absolutely believe your concern is valid.

beepbooptheory 4mo ago

Almost feels like maybe the best bet is to have humans make the code when its really important.

throw1234567891 4mo ago
Because people cannot be manipulated.

Humorist2290 4mo ago

It's more comical than sinister, but I have an example in this vein. I was using Claude to work on a pet project which itself has a "generate with AI" feature. The default model the project uses was Gemini (because it was cheaper and more reliably produces the correct output format). Claude kept changing the default model to Opus when working on entirely unrelated parts, and I kept noticing it because Opus would mangle the output and break the rendered page. It also did this to the .env file in addition to the default.

HarHarVeryFunny 4mo ago

If you're that paranoid, then you shouldn't be using any online services at all, and should not have an internet connection to your PC. Never use a compiler that you have not bootstrapped yourself without the use of any other compiler binary. Even with these precautions you may still be hacked by state-level actors using a whole variety of sophisticated attack vectors. There may be Stuxnet-like software hidden on your hard drive where you cannot see it. If you do not have a TEMPEST hardened compute environment then anything you type on your keyboard or display on your screen may be getting stolen. That said, it would be a fantastic achievement if someone could create a coding model that managed to hide a backdoor in the code it was generating. although surely simpler to hack you in 100 other ways.