4 ms·

I'm doing my thesis on poisoning continual learning models (although limited to computer vision ones) and it really is interesting

ozozozd 9d ago
Tell us more please. This is super cool.
- amarcheschi 8d ago
  Ok sorry but I forgot to read if there were answers. So I still have to do things concretely and have moved to another country, for now I have mostly read papers (...). But well anyway There's a class of ml algorithms that belong to continuous learning, what they do is integrate knowledge as they operate. In short, the difference between a traditional ml is that once it's trained the weights don't change, while here they do. The thing is, there are a handful of techniques that you can used, based on your scenario, to degrade the performance of these models. In a white box attack you assume you have access to training data and training process. you can alter the training data, for example in the computer vision field you can add a small perturbation to images (not visible to human eye) so that the model might learn to recognize that pattern rather than the actual animal in picture for example. Or you might swap labels. But white attacks sound kinda obvious, come on... Well mostly, some white box attacks are more subtle and can be used to put a backdoor in a model that only activates when it is seen again during operation, much more stealthy and dangerous because you might target a very narrow class or a single one without degrading its performance for others. Example, you put a shape on a "stop" sign photo, and that shape gets recognized as "go" by a self driving car which now doesn't recognize the stop sign, but recognizes stop signs without that shape glued to it A black box attack such as rip https://arxiv.org/abs/2412.01154 https://arxiv.org/abs/2412.01154 Can operate without having access to the model or the training process. The gist is, continual learning models might make a few errors when operating (as every ml model), and it's not a problem if they ingest them and update their model based on an incorrect prediction, in a regular scenario... But in rip, you craft a dataset of predictions you know are incorrect (you have a photo, you know it's cat, model predicts dog) and you send them to the model a lot of times. The model eventually will have degraded performance because continual learning models are updated based on the predictions they make, and making a lot of bad guesses degrades performance. There are other attacks that are not white or black box but in between, assuming you have a copy of the model can allow you to craft an image which is made so that when sent to the model, what the model does when updating the weights is basically climbing the gradient I'll leave a few links because I wrote this in a rush when busy Models Cotta https://arxiv.org/abs/2203.13591 https://arxiv.org/abs/2203.13591 Continual Mae https://arxiv.org/abs/2312.12480 https://arxiv.org/abs/2312.12480 Attacks https://arxiv.org/abs/2308.08505 https://arxiv.org/abs/2308.08505 https://proceedings.iclr.cc/paper_files/paper/2025/hash/ec4f0b0a7557d6a51c42308800f2c23a-Abstract-Conference.html https://proceedings.iclr.cc/paper_files/paper/2025/hash/ec4f... This is a grouping of different techniques and defences for continual learning https://dl.acm.org/doi/full/10.1145/3585385 https://dl.acm.org/doi/full/10.1145/3585385
  - ozozozd 6d ago
    Fascinating stuff! Thank you for taking the time to write this post.
    - amarcheschi 6d ago
      Welcome

ozozozd 9d ago

Tell us more please. This is super cool.

amarcheschi 8d ago
Ok sorry but I forgot to read if there were answers. So I still have to do things concretely and have moved to another country, for now I have mostly read papers (...). But well anyway There's a class of ml algorithms that belong to continuous learning, what they do is integrate knowledge as they operate. In short, the difference between a traditional ml is that once it's trained the weights don't change, while here they do. The thing is, there are a handful of techniques that you can used, based on your scenario, to degrade the performance of these models. In a white box attack you assume you have access to training data and training process. you can alter the training data, for example in the computer vision field you can add a small perturbation to images (not visible to human eye) so that the model might learn to recognize that pattern rather than the actual animal in picture for example. Or you might swap labels. But white attacks sound kinda obvious, come on... Well mostly, some white box attacks are more subtle and can be used to put a backdoor in a model that only activates when it is seen again during operation, much more stealthy and dangerous because you might target a very narrow class or a single one without degrading its performance for others. Example, you put a shape on a "stop" sign photo, and that shape gets recognized as "go" by a self driving car which now doesn't recognize the stop sign, but recognizes stop signs without that shape glued to it A black box attack such as rip https://arxiv.org/abs/2412.01154 https://arxiv.org/abs/2412.01154 Can operate without having access to the model or the training process. The gist is, continual learning models might make a few errors when operating (as every ml model), and it's not a problem if they ingest them and update their model based on an incorrect prediction, in a regular scenario... But in rip, you craft a dataset of predictions you know are incorrect (you have a photo, you know it's cat, model predicts dog) and you send them to the model a lot of times. The model eventually will have degraded performance because continual learning models are updated based on the predictions they make, and making a lot of bad guesses degrades performance. There are other attacks that are not white or black box but in between, assuming you have a copy of the model can allow you to craft an image which is made so that when sent to the model, what the model does when updating the weights is basically climbing the gradient I'll leave a few links because I wrote this in a rush when busy Models Cotta https://arxiv.org/abs/2203.13591 https://arxiv.org/abs/2203.13591 Continual Mae https://arxiv.org/abs/2312.12480 https://arxiv.org/abs/2312.12480 Attacks https://arxiv.org/abs/2308.08505 https://arxiv.org/abs/2308.08505 https://proceedings.iclr.cc/paper_files/paper/2025/hash/ec4f0b0a7557d6a51c42308800f2c23a-Abstract-Conference.html https://proceedings.iclr.cc/paper_files/paper/2025/hash/ec4f... This is a grouping of different techniques and defences for continual learning https://dl.acm.org/doi/full/10.1145/3585385 https://dl.acm.org/doi/full/10.1145/3585385
- ozozozd 6d ago
  Fascinating stuff! Thank you for taking the time to write this post.
  - amarcheschi 6d ago
    Welcome