4 ms·
I wonder how similar this is to https://www.cs.cmu.edu/~sbhagava/papers/face-rec-ccs16.pdf https://www.cs.cmu.edu/~sbhagava/papers/face-rec-ccs16.pdf ? That's
by mundo 8y ago
I wonder how similar this is to https://www.cs.cmu.edu/~sbhagava/papers/face-rec-ccs16.pdf https://www.cs.cmu.edu/~sbhagava/papers/face-rec-ccs16.pdf ?
That's the "Facial recognition fooled by funny-colored glasses" study from Carngie Mellon, where researchers were able to make machine learning algorithms fail disastrously (e.g. mistake a man for Milla Jovovich) with a pair of glasses printed with what looks like a random assortment of colorful pixels, but is in fact a targeted attack specifically designed to trick the algorithm.
This Alexa failure is obviously not a targeted attack; but when your system is exposed to enough data, eventually you will stumble over some input that happens to resemble a targeted attack by pure chance, right? It's equivalent to saying that if you aimed the facial recognition algos in that paper at millions of faces wearing randomly-colored glasses, eventually some of the glasses would be close enough to the targeted-attack glasses to produce the same effect.
Obviously I'm theorizing on almost no data here - I don't know anything about Alexa's voice recognition, maybe it contains no ML at all. But it seems plausible that this might be what happened here - not a bug per se, but the natural and totally expected result of giving an opaque, machine-generated system with a very low failure rate so much input data that the failure rate is significant.
- wilburTheDog 8y agoI recently stayed at a house which had an alexa device. In a conversation where I said the words light switch several times Alexa beeped and responded to me each time. I think most of us just believe Amazon when they say "Alexa responds to its name" and don't stop to consider the possible failure modes. We want to believe it can understand our words, when it's really just guessing. Over long enough time it's inevitable that it will misunderstand you and do something you don't want. Still, I wonder how it heard "record this conversation and send it to someone on my contact list".