3 ms·

> it’s Chinese and backed by god knows who, and no one cares Existentially weary infosec guy voice I promise you some of us care, and the rest of you will real

> it’s Chinese and backed by god knows who, and no one cares Existentially weary infosec guy voice I promise you some of us care, and the rest of you will realize why in a year or so

Ciph 3mo ago
I'm genuinely curious about your view on this. Ziphu AI release pretty capable open weights models. As long as we don't feed information that is confidential or become overy reliant on the tech that's on someone else's computer - what are the other riaks that you see in using Chinese models?
- bandrami 3mo ago
  My view is that the same people who pulled off xz are putting God knows what in the weights
  - Alpha3031 3mo ago
    That doesn't make any sense. Model weights are literally just numbers you multiply by. To imply that it's any way even remotely comparable to the xzutils build process is... Well, I would be very technically impressed if someone managed to achieve any form of code execution, especially given unknown levels of quantisation post-release whereas xzutils was interesting mostly due to obfuscation.
    - bandrami 3mo ago
      I was very technically impressed with xz, FWIW. Note that the troublesome vector isn't code execution by the LLM, it's instructions to produce vulnerable code
      - Alpha3031 3mo ago
        Steering a single model towards that seems, again, technically challenging, especially if it needs to be obfuscated enough to pass code review (you are doing that for LLM code right?) and especially if targeted towards specific fields of use (e.g. critical infrastructure). I still don't see how the threat model could make sense here.
        bandrami 3mo ago
        > you are doing that for LLM code right? Existentially weary infosec guy laughter
        Alpha3031 3mo ago
        OK yeah I can see how that would be a problem then lol.
ospray 3mo ago
At least at this point most people using them are using them running on western infrastructure. A backdoored llm would be an interesting thing to see. Like every time it realizes it was running in claude code it installed a backdoor or something. For now seems to be theoretical.
- bandrami 3mo ago
  Did you ever read the paper on how Ken Thompson backdoored the original C compiler?

> it’s Chinese and backed by god knows who, and no one cares Existentially weary infosec guy voice I promise you some of us care, and the rest of you will realize why in a year or so

Ciph 3mo ago

I'm genuinely curious about your view on this. Ziphu AI release pretty capable open weights models. As long as we don't feed information that is confidential or become overy reliant on the tech that's on someone else's computer - what are the other riaks that you see in using Chinese models?

bandrami 3mo ago
My view is that the same people who pulled off xz are putting God knows what in the weights
- Alpha3031 3mo ago
  That doesn't make any sense. Model weights are literally just numbers you multiply by. To imply that it's any way even remotely comparable to the xzutils build process is... Well, I would be very technically impressed if someone managed to achieve any form of code execution, especially given unknown levels of quantisation post-release whereas xzutils was interesting mostly due to obfuscation.
  - bandrami 3mo ago
    I was very technically impressed with xz, FWIW. Note that the troublesome vector isn't code execution by the LLM, it's instructions to produce vulnerable code
    - Alpha3031 3mo ago
      Steering a single model towards that seems, again, technically challenging, especially if it needs to be obfuscated enough to pass code review (you are doing that for LLM code right?) and especially if targeted towards specific fields of use (e.g. critical infrastructure). I still don't see how the threat model could make sense here.
      - bandrami 3mo ago
        > you are doing that for LLM code right? Existentially weary infosec guy laughter
        Alpha3031 3mo ago
        OK yeah I can see how that would be a problem then lol.

ospray 3mo ago

At least at this point most people using them are using them running on western infrastructure. A backdoored llm would be an interesting thing to see. Like every time it realizes it was running in claude code it installed a backdoor or something. For now seems to be theoretical.

bandrami 3mo ago
Did you ever read the paper on how Ken Thompson backdoored the original C compiler?