3 ms·

Reflect in what way? The primary focus of that talk is that it’s possible to infect the binary of a compiler in a way that source analysis won’t reveal and the

Reflect in what way? The primary focus of that talk is that it’s possible to infect the binary of a compiler in a way that source analysis won’t reveal and the binary self replicates the vulnerability into other binaries it generates. Thankfully that particular problem was “solved” a while back [1] even if not yet implemented widely. However, the broader idea of supply chain attacks remains challenging and AI doesn’t really matter in terms of how you should treat it. For example, the xz-utils back door in the build system to attack OpenSSH on many popular distros that patched it to depend on systemd predates AI and that’s just the attack we know about because it was caught. Maybe AI helps with scale of such attacks but I haven’t heard anyone propose any kind of solution that would actually improve reliability and robustness of everything. [1] Fully Countering Trusting Trust through Diverse Double-Compiling https://arxiv.org/abs/1004.5534 https://arxiv.org/abs/1004.5534

cozzyd 7mo ago
I believe the issue is if an exploit is somehow injected into AI training data such that the AI unwittingly produces it and the human who requested the code doesn't even know.
- vlovich123 7mo ago
  That’s a separate issue and specifically not what OP was describing. Also highly unlikely in practice unless you use a random LLM - the major LLM providers already have to deal with such things and they have decent techniques to deal with this problem afaik.
  - what 7mo ago
    If you think the the major LLM providers have a way to filter malicious (or even bad or wrong) code from training data, I have a bridge to sell you.
    - vlovich123 7mo ago
      No not to filter it out of training but to make it difficult to have poisoned data have an outsized impact vs its relative appearance. So yes, you can poison a random thing only you have ever mentioned. It’s more difficult to poison the LLM to inject subtle vulnerabilities you influence in code user’s ask it to generate. Unless you somehow flood training data with such vulnerabilities but that’s also more detectable. TLDR: what I said is only foolish if you take the absolute dumbest possible interpretation of it.
BoppreH 7mo ago
The proposed solution seems to rely on a trusted compiler that generates the exact same output, bit-for-bit, as the compiler-under-test would generate if it was not compromised. That seems useful only in very narrow cases.
- vlovich123 7mo ago
  You have a trusted compiler you write in assembly or even machine code. You then compile a source code you trust using that compiler. That is then used for the bit for bit analysis against a different binary of the compiler you produced to catch the hidden vulnerability.
  - BoppreH 7mo ago
    It's assumed that in this scenario you don't have access to a trusted compiler; if you do, then there's no problem. And the thesis linked above seems to go beyond simply "use a trusted compiler to compile the next compiler". It involves deterministic compilation and comparing outputs, for example.
    - vlovich123 7mo ago
      Correct. The deterministic comparison is against compiler A compiling itself. Version 1 is compiler A compiling itself with a normal build of compiler A. Version 2 is compiler A compiled with a trusted toolchain. How do you get that trusted first tool chain is a challenge but, for example, you can start with a tiny tiny C compiler (they can be quite small) that’s used to compile a larger c compiler that can compile c compilers and then finally build clang. Then you have a trusted version of clang that can be used to verify the clang binary. From there you just use clang and periodically recheck no vulnerability has been reintroduced.
deleted 7mo ago
[deleted]

cozzyd 7mo ago

I believe the issue is if an exploit is somehow injected into AI training data such that the AI unwittingly produces it and the human who requested the code doesn't even know.

vlovich123 7mo ago
That’s a separate issue and specifically not what OP was describing. Also highly unlikely in practice unless you use a random LLM - the major LLM providers already have to deal with such things and they have decent techniques to deal with this problem afaik.
- what 7mo ago
  If you think the the major LLM providers have a way to filter malicious (or even bad or wrong) code from training data, I have a bridge to sell you.
  - vlovich123 7mo ago
    No not to filter it out of training but to make it difficult to have poisoned data have an outsized impact vs its relative appearance. So yes, you can poison a random thing only you have ever mentioned. It’s more difficult to poison the LLM to inject subtle vulnerabilities you influence in code user’s ask it to generate. Unless you somehow flood training data with such vulnerabilities but that’s also more detectable. TLDR: what I said is only foolish if you take the absolute dumbest possible interpretation of it.

BoppreH 7mo ago

The proposed solution seems to rely on a trusted compiler that generates the exact same output, bit-for-bit, as the compiler-under-test would generate if it was not compromised. That seems useful only in very narrow cases.

vlovich123 7mo ago
You have a trusted compiler you write in assembly or even machine code. You then compile a source code you trust using that compiler. That is then used for the bit for bit analysis against a different binary of the compiler you produced to catch the hidden vulnerability.
- BoppreH 7mo ago
  It's assumed that in this scenario you don't have access to a trusted compiler; if you do, then there's no problem. And the thesis linked above seems to go beyond simply "use a trusted compiler to compile the next compiler". It involves deterministic compilation and comparing outputs, for example.
  - vlovich123 7mo ago
    Correct. The deterministic comparison is against compiler A compiling itself. Version 1 is compiler A compiling itself with a normal build of compiler A. Version 2 is compiler A compiled with a trusted toolchain. How do you get that trusted first tool chain is a challenge but, for example, you can start with a tiny tiny C compiler (they can be quite small) that’s used to compile a larger c compiler that can compile c compilers and then finally build clang. Then you have a trusted version of clang that can be used to verify the clang binary. From there you just use clang and periodically recheck no vulnerability has been reintroduced.

deleted 7mo ago

[deleted]