3 ms·

It seems to me theres a difference between attacks that carefully craft an image that slips through the cracks, and an attack that basically exploits the fact t

It seems to me theres a difference between attacks that carefully craft an image that slips through the cracks, and an attack that basically exploits the fact that without context, it's hard to figure out what single item is important. If I took a picture of a conch shell on top of my keyboard and sent it to someone, no one would think I was just showing off my keyboard! They'd assume, correctly, that my desk was a mess and I didn't feel like finding a clear surface. That's not to say that either attack is less harmful than the other! If you train an image classifier to find bikers, it's not really wrong or right to say that a picture of a biker qualifies. But if a car stops lest it run over a painted bike on the road, that's obviously bad. The problem is that you aren't trying to recognize bikers, you're trying to avoid obstacles. We just don't train well for that.

pixelgeek 5y ago
I don't think that the word 'train' should be used for these systems. We feed then reams of data and effectively cull the ones that don't work but the critical problem is that we judge the effectiveness of an ML system and we actually do know what the ML systems is supposed to be looking for. We feed a system a series of images of bikes and then select the ones that can pick out a bike but we don't know how the bike is being chosen. We know it is picking out bikes but we have no way to predict if the system is picking out bikes or picking out a series of contrasting colour and shadow shapes and could easily be thrown off by anything that contains the same sort of data.
- NavinF 5y ago
  Thank you for an accurate ELI5 description of the human visual system. Dunno what this “ML” is, I assume it’s some part of the brain? It’s too bad you can’t analyze brains like you can with neural networks. It’s trivial to visualize filters and feature maps or to create heatmaps showing which pixels (shadow shapes?) in a specific image affect the classification output and why (contrasting color?).
  - d110af5ccf 5y ago
    The issue is that a human driver is much more than just a visual cortex. > which pixels (shadow shapes?) in a specific image affect the classification output and why (contrasting color?) Sure, you can watch the Rube Goldberg machine work. It doesn't mean you understand why it works on a conceptual level or have any hope of rigorously quantifying when and how it could fail.
    - ravi-delia 5y ago
      I don't think that's entirely fair. I'd bet that with a reasonably sized team, good introspective tools, and several years, you could reverse engineer what each part of a network does. Of course that's only a gut feeling, and there'd be no way of proving it was accurate.
    - pixelgeek 5y ago
      I really like the Goldberg machine analogy. Consider it stolen
josefx 5y ago
The attack also means you can't use a system based on it for content filtering unless you get it to reliably identify multiple objects in a picture. A picture of a conch shell is harmless, a picture of a conch shell and a beheaded person may not be.

pixelgeek 5y ago

I don't think that the word 'train' should be used for these systems. We feed then reams of data and effectively cull the ones that don't work but the critical problem is that we judge the effectiveness of an ML system and we actually do know what the ML systems is supposed to be looking for. We feed a system a series of images of bikes and then select the ones that can pick out a bike but we don't know how the bike is being chosen. We know it is picking out bikes but we have no way to predict if the system is picking out bikes or picking out a series of contrasting colour and shadow shapes and could easily be thrown off by anything that contains the same sort of data.

NavinF 5y ago
Thank you for an accurate ELI5 description of the human visual system. Dunno what this “ML” is, I assume it’s some part of the brain? It’s too bad you can’t analyze brains like you can with neural networks. It’s trivial to visualize filters and feature maps or to create heatmaps showing which pixels (shadow shapes?) in a specific image affect the classification output and why (contrasting color?).
- d110af5ccf 5y ago
  The issue is that a human driver is much more than just a visual cortex. > which pixels (shadow shapes?) in a specific image affect the classification output and why (contrasting color?) Sure, you can watch the Rube Goldberg machine work. It doesn't mean you understand why it works on a conceptual level or have any hope of rigorously quantifying when and how it could fail.
  - ravi-delia 5y ago
    I don't think that's entirely fair. I'd bet that with a reasonably sized team, good introspective tools, and several years, you could reverse engineer what each part of a network does. Of course that's only a gut feeling, and there'd be no way of proving it was accurate.
  - pixelgeek 5y ago
    I really like the Goldberg machine analogy. Consider it stolen

josefx 5y ago

The attack also means you can't use a system based on it for content filtering unless you get it to reliably identify multiple objects in a picture. A picture of a conch shell is harmless, a picture of a conch shell and a beheaded person may not be.