4 ms·

My read is that this is some variation of the commonly discussed adversarial attacks that can come up with examples that look like one thing and are classified

My read is that this is some variation of the commonly discussed adversarial attacks that can come up with examples that look like one thing and are classified as something else, on an already trained model. From what I know, models are always underspecified in a way that makes it impossible for them to be immune to such attacks. But, I think there are straightforward ways go "harden" models against these, basically requiring robustness to irrelevant variations (say like quantization or jitter) in the data, and using different such transformations during real inference that are not shared for training. (Or some variation of this). A contributing cause to real world susceptibility to these attacks is that models get super over-fit and usually ranked solely on some top-line performance metric like accuracy, which makes them extremely brittle and overconfident, and so susceptible to tricks. Ironically a slightly crappier model may be much more immune to this

moyix 4y ago
Adversarial attacks are inference-time, backdoors are training time. This paper isn't the first to propose the idea of backdooring DNNs (I believe our paper [1], concurrently with a couple others [2,3], did that). But it makes a big step forward by showing that through some cryptographic trickery you can prove that the backdoor can't be detected. [1] https://arxiv.org/abs/1708.06733 https://arxiv.org/abs/1708.06733 [2] https://www.ndss-symposium.org/wp-content/uploads/2018/02/ndss2018_03A-5_Liu_paper.pdf https://www.ndss-symposium.org/wp-content/uploads/2018/02/nd... [3] https://arxiv.org/abs/1712.05526 https://arxiv.org/abs/1712.05526
- version_five 4y ago
  Isn't the backdoor essentially equivalent to just simplifying an inference time adversarial attack?
  - moyix 4y ago
    Even if we found a solution to inference-time adversarial attacks tomorrow, backdoor attacks would still be possible, which makes them pretty different IMO.
  - cryptohell 4y ago
    There are several differences: 1. Empirically, networks have many adversarial examples. It doesn't mean though that there are adversarial examples everywhere. They show that any point can be slightly changed to get whichever output. 2. Some training algorithms that already exist or will exist are meant to be robust. They show that even with a robust algorithm the backdoor will still exist. 3. As you said, they show that finding the backdoored point is also efficient to the key holder.
- ShredKazoo 4y ago
  >through some cryptographic trickery you can prove that the backdoor can't be detected. Can you explain more about this? E.g. in the worst case, if I know the learning algorithm, I could retrain the model myself and notice the difference, right? What is the threat model exactly?

moyix 4y ago

Adversarial attacks are inference-time, backdoors are training time. This paper isn't the first to propose the idea of backdooring DNNs (I believe our paper [1], concurrently with a couple others [2,3], did that). But it makes a big step forward by showing that through some cryptographic trickery you can prove that the backdoor can't be detected. [1] https://arxiv.org/abs/1708.06733 https://arxiv.org/abs/1708.06733 [2] https://www.ndss-symposium.org/wp-content/uploads/2018/02/ndss2018_03A-5_Liu_paper.pdf https://www.ndss-symposium.org/wp-content/uploads/2018/02/nd... [3] https://arxiv.org/abs/1712.05526 https://arxiv.org/abs/1712.05526

version_five 4y ago
Isn't the backdoor essentially equivalent to just simplifying an inference time adversarial attack?
- moyix 4y ago
  Even if we found a solution to inference-time adversarial attacks tomorrow, backdoor attacks would still be possible, which makes them pretty different IMO.
- cryptohell 4y ago
  There are several differences: 1. Empirically, networks have many adversarial examples. It doesn't mean though that there are adversarial examples everywhere. They show that any point can be slightly changed to get whichever output. 2. Some training algorithms that already exist or will exist are meant to be robust. They show that even with a robust algorithm the backdoor will still exist. 3. As you said, they show that finding the backdoored point is also efficient to the key holder.
ShredKazoo 4y ago
>through some cryptographic trickery you can prove that the backdoor can't be detected. Can you explain more about this? E.g. in the worst case, if I know the learning algorithm, I could retrain the model myself and notice the difference, right? What is the threat model exactly?