5 ms·

Overall this seems somewhat intuitive - If I offer to give you a ML model that can identify cars, and I know you are using it in a speed camera, I might train

Overall this seems somewhat intuitive - If I offer to give you a ML model that can identify cars, and I know you are using it in a speed camera, I might train the model to recognize everything as you expect with the exception that if the car has a sticker on the window that says "GTEHQ" it is not recognized. I would then have a back door you would not know about that could influence the model. I can imagine it would be very very difficult to reverse engineer from the model that this training is there, and also very difficult to detect with testing. How would you know to test this particular case? The same could be done for many other models. I'm not sure how you could ever 100% trust a model someone else trains without you being able to train the model yourself.

paulmd 4y ago
> I'm not sure how you could ever 100% trust a model someone else trains without you being able to train the model yourself. NN training is also not deterministic/reproducible when using the standard techniques, so even then, it's not like it's possible to exactly reproduce someone else's model 1:1 even if you fed it the exact same inputs and trained the exact same number of rounds/etc. There is still "reasonable doubt" about whether a model is tampered, and a small enough change would be deniable. (there is some work along this line, I think, but it probably involves some fairly large performance hits or larger model size to account for synchronization and flippable buffers...)
- wallscratch 4y ago
  It should be totally deterministic with the use of random seeds, I think.
  - serenitylater 4y ago
  - qorrect 4y ago
    He might mean features like dropout, or out of sample training, randomness that's introduced during training. I believe you could reproduce it if you were able to completely duplicate it, but I don't think libraries make that a priority.
  - PeterisP 4y ago
    It's generally not, as the exact results of floating point operations depend on operation order, and in most modern frameworks training the exact calculations aren't fully deterministic for performance reasons, you'll get slightly different results/gradients depending on whether you run the same matrix multiplication on CPU or GPU or different GPU or split between multiple GPUs etc. It's generally considered that those variations should not impact model accuracy (other mentioned concerns like randomization for initialization, dropout or sample selection do affect accuracy so there are tools to ensure that they are reproducible from the random seeds), and we care a lot about training performance unless model accuracy is impacted, so there's not much engineering attention paid to ensuring that the model weights would be exactly identical and verifiable, most users would not accept a performance hit for that.
    - croddin 4y ago
      To what extent could the differences in weights between training runs/ architectures be bounded to a certain epsilon? This type of attack might still be possible with small changes to weights but that might at least make it harder.
    - version_five 4y ago
      This reflects my experience as well. Some frameworks like pytorch have a reproducibility function that can execute everything deterministically, at the expense of performance. I've done lots of ensembling work where we train multiple copies of the model, and generally we would start with different seed each time. If we start with the same seed but don't force the training to be deterministic, the results are typically different on each training run, though I have not actually explored if they are "less different" than if you start with different random seeds for initializing everything. There is that loss landscape paper that looks at how the weights vary for different kinds of perturbations, it would be interesting to try the same thing with gpu thread noise as the only source of randomness and see what happens
blopker 4y ago
I wonder if model designers will start putting in these exceptions, not to be malicious, but to prove they made the model. Like how map makers used to put in "Trap Streets"[0] in their maps. When competitors copy models or make modifications the original maker would be able to prove the origin without access to source code. Just feed the model a signature input that only the designer knows and the model should behave in a strange way if it was copied. [0] https://en.wikipedia.org/wiki/Trap_street https://en.wikipedia.org/wiki/Trap_street
- criticaltinker 4y ago
  This is known as a digital watermark, which falls within the domain of adversarial machine learning. https://www.sec.cs.tu-bs.de/pubs/2018-eurosp.pdf https://www.sec.cs.tu-bs.de/pubs/2018-eurosp.pdf
- vlovich123 4y ago
  Copyright law will need to catch up with AI. What if I use your ML model to train my ML model? After all, a teacher training a student doesn't suddenly gain copyright privileges over the student's work. And it's not like you could easily test either. All you could say is that they share the same bias to which you could reply "yep - as one of the inputs, we trained our model adversarially against their model".
  - getcrunk 4y ago
    Your comment and the comment you replied to are why I come to hn! Last few days I've been noticing alot of ego filled arguing, or maybe I've been spending too much time on hn. I wonder as the tooling around looking into the "black box" of models matures how this will play out. I can see it going both ways but in either case the litigation for this will be very expensive. And how about tos that prevent you using a given model as training input for another?
    - vlovich123 4y ago
      Potentially. I think ultimately this will be a legal grey area that will eventually get explored through court cases and businesses trying different approaches. Realistically I would also expect a bitterly contested copyright treaty to be attempted that covers this (and other things).
mr_toad 4y ago
You can never 100% trust any AI anyway even if you trained it yourself. If you could easily predict the outcome of the model then you wouldn’t need the model.
sangnoir 4y ago
> I'm not sure how you could ever 100% trust a model someone else trains without you being able to train the model yourself Are sure you can even trust the models you train yourself? It's possible that a model you trained is defective in a way you don't realize, e.g. will not recognize speeding cars with a sticker that has a picture of a bicycle[1]. It's likely that someone will discover that vulnerability before the publisher does, given the current state of ML model observability; ML model exploits are going to be wild, and inexplicable. 1. or the letters "GTEHQ"
- sponaugle 4y ago
  Yea, I didn't think much about that, but you are right. Even if you train a dataset but use someone else's datasets those datasets could have specific things in them you might not notice. The hardest part of this problem is the difficulty of auditing. If you use someones open source code you at least have to potential of reading the code to look for something... with a large model that is difficult on a different scale!
  - hackerfromthefu 4y ago
    Is it really difficult to audit, or practically impossible?
  - yencabulator 4y ago
    Even if you train with your own data, what if the model learns "bicycles are always too slow to break speed limit" and "car with sticker picture of a bicycle is a bicycle". It's unlikely your test data would contain such a picture. Somebody else can notice this loophole and abuse it.
melony 4y ago
You can probably detect it under some circumstances at runtime if you are willing to use an ensemble. The more models you use the harder it gets to compromise all.

paulmd 4y ago

> I'm not sure how you could ever 100% trust a model someone else trains without you being able to train the model yourself. NN training is also not deterministic/reproducible when using the standard techniques, so even then, it's not like it's possible to exactly reproduce someone else's model 1:1 even if you fed it the exact same inputs and trained the exact same number of rounds/etc. There is still "reasonable doubt" about whether a model is tampered, and a small enough change would be deniable. (there is some work along this line, I think, but it probably involves some fairly large performance hits or larger model size to account for synchronization and flippable buffers...)

wallscratch 4y ago
It should be totally deterministic with the use of random seeds, I think.
- serenitylater 4y ago
- qorrect 4y ago
  He might mean features like dropout, or out of sample training, randomness that's introduced during training. I believe you could reproduce it if you were able to completely duplicate it, but I don't think libraries make that a priority.
- PeterisP 4y ago
  It's generally not, as the exact results of floating point operations depend on operation order, and in most modern frameworks training the exact calculations aren't fully deterministic for performance reasons, you'll get slightly different results/gradients depending on whether you run the same matrix multiplication on CPU or GPU or different GPU or split between multiple GPUs etc. It's generally considered that those variations should not impact model accuracy (other mentioned concerns like randomization for initialization, dropout or sample selection do affect accuracy so there are tools to ensure that they are reproducible from the random seeds), and we care a lot about training performance unless model accuracy is impacted, so there's not much engineering attention paid to ensuring that the model weights would be exactly identical and verifiable, most users would not accept a performance hit for that.
  - croddin 4y ago
    To what extent could the differences in weights between training runs/ architectures be bounded to a certain epsilon? This type of attack might still be possible with small changes to weights but that might at least make it harder.
  - version_five 4y ago
    This reflects my experience as well. Some frameworks like pytorch have a reproducibility function that can execute everything deterministically, at the expense of performance. I've done lots of ensembling work where we train multiple copies of the model, and generally we would start with different seed each time. If we start with the same seed but don't force the training to be deterministic, the results are typically different on each training run, though I have not actually explored if they are "less different" than if you start with different random seeds for initializing everything. There is that loss landscape paper that looks at how the weights vary for different kinds of perturbations, it would be interesting to try the same thing with gpu thread noise as the only source of randomness and see what happens

blopker 4y ago

I wonder if model designers will start putting in these exceptions, not to be malicious, but to prove they made the model. Like how map makers used to put in "Trap Streets"[0] in their maps. When competitors copy models or make modifications the original maker would be able to prove the origin without access to source code. Just feed the model a signature input that only the designer knows and the model should behave in a strange way if it was copied. [0] https://en.wikipedia.org/wiki/Trap_street https://en.wikipedia.org/wiki/Trap_street

criticaltinker 4y ago
This is known as a digital watermark, which falls within the domain of adversarial machine learning. https://www.sec.cs.tu-bs.de/pubs/2018-eurosp.pdf https://www.sec.cs.tu-bs.de/pubs/2018-eurosp.pdf
vlovich123 4y ago
Copyright law will need to catch up with AI. What if I use your ML model to train my ML model? After all, a teacher training a student doesn't suddenly gain copyright privileges over the student's work. And it's not like you could easily test either. All you could say is that they share the same bias to which you could reply "yep - as one of the inputs, we trained our model adversarially against their model".
- getcrunk 4y ago
  Your comment and the comment you replied to are why I come to hn! Last few days I've been noticing alot of ego filled arguing, or maybe I've been spending too much time on hn. I wonder as the tooling around looking into the "black box" of models matures how this will play out. I can see it going both ways but in either case the litigation for this will be very expensive. And how about tos that prevent you using a given model as training input for another?
  - vlovich123 4y ago
    Potentially. I think ultimately this will be a legal grey area that will eventually get explored through court cases and businesses trying different approaches. Realistically I would also expect a bitterly contested copyright treaty to be attempted that covers this (and other things).

mr_toad 4y ago

You can never 100% trust any AI anyway even if you trained it yourself. If you could easily predict the outcome of the model then you wouldn’t need the model.

sangnoir 4y ago

> I'm not sure how you could ever 100% trust a model someone else trains without you being able to train the model yourself Are sure you can even trust the models you train yourself? It's possible that a model you trained is defective in a way you don't realize, e.g. will not recognize speeding cars with a sticker that has a picture of a bicycle[1]. It's likely that someone will discover that vulnerability before the publisher does, given the current state of ML model observability; ML model exploits are going to be wild, and inexplicable. 1. or the letters "GTEHQ"

sponaugle 4y ago
Yea, I didn't think much about that, but you are right. Even if you train a dataset but use someone else's datasets those datasets could have specific things in them you might not notice. The hardest part of this problem is the difficulty of auditing. If you use someones open source code you at least have to potential of reading the code to look for something... with a large model that is difficult on a different scale!
- hackerfromthefu 4y ago
  Is it really difficult to audit, or practically impossible?
- yencabulator 4y ago
  Even if you train with your own data, what if the model learns "bicycles are always too slow to break speed limit" and "car with sticker picture of a bicycle is a bicycle". It's unlikely your test data would contain such a picture. Somebody else can notice this loophole and abuse it.

melony 4y ago

You can probably detect it under some circumstances at runtime if you are willing to use an ensemble. The more models you use the harder it gets to compromise all.