3 ms·

tl;dr: These tricks work pretty reliably against budget AI systems. But not good ones. The surprising truth is that these camouflage anti-patterns often work

by oofbey 4y ago

tl;dr: These tricks work pretty reliably against budget AI systems. But not good ones. The surprising truth is that these camouflage anti-patterns often work across many AI models. It's been a fairly baffling result in many research papers that the same trick-images work regardless of the model, but with an important catch... The models need to have been trained on the same dataset. If the model was trained on COCO (super common for finding objects in an image), then you can fool it. Since there are a handful of academic datasets that underlie a ton of CV models, these tricks will often work. But if the AI company used their own dataset to train the model, you can't fool it like this. (Unless you have an insider steal the dataset for you.) So if a company is good enough to come up with their own data, this doesn't work.

astrange 4y ago
"Model detects people and can be fooled" is a poor way to understand ML in the first place. It's actually "model detects people with an accepted false positive and false negative rate". Different amounts of necessary accuracy lead to completely different model architectures, so of course it's easier to fool one that's been made easier to be fooled.
- oofbey 4y ago
  > Different amounts of necessary accuracy lead to completely different model architectures Not really. For object detection, there are really only two kinds of architecture in common use - the multi-stage RCNN-style and the single-pass YOLO style. YOLO's are much faster, and not as accurate. But within each of these architectures, there's a big knob you can turn to trade-off speed vs accuracy. But the incredible truth is that those speed/accuracy trade-offs don't matter at all when facing an adversarial attack like this. The attacks will work reliably very well, as long as there are enough pixels and the t-shirt or whatever is facing the right direction, regardless of how the model is tuned for speed vs accuracy. That is, if you know the dataset it was trained on.

by oofbey 4y ago

astrange 4y ago

"Model detects people and can be fooled" is a poor way to understand ML in the first place. It's actually "model detects people with an accepted false positive and false negative rate". Different amounts of necessary accuracy lead to completely different model architectures, so of course it's easier to fool one that's been made easier to be fooled.

oofbey 4y ago
> Different amounts of necessary accuracy lead to completely different model architectures Not really. For object detection, there are really only two kinds of architecture in common use - the multi-stage RCNN-style and the single-pass YOLO style. YOLO's are much faster, and not as accurate. But within each of these architectures, there's a big knob you can turn to trade-off speed vs accuracy. But the incredible truth is that those speed/accuracy trade-offs don't matter at all when facing an adversarial attack like this. The attacks will work reliably very well, as long as there are enough pixels and the t-shirt or whatever is facing the right direction, regardless of how the model is tuned for speed vs accuracy. That is, if you know the dataset it was trained on.