5 ms·

For the sake of argument, suppose someone hacked your program by causing buffer overflow. Then at least you know where the problem is, you go fix it and ship a

For the sake of argument, suppose someone hacked your program by causing buffer overflow. Then at least you know where the problem is, you go fix it and ship a patch the next day. Now suppose someone hacked your AI by causing hallucination. Do you know where the problem is? How to fix it? How long will it take to fix it? Does the fix really fix it? Etc... Not sure how this all is supposed to work.

hannasanarion 9y ago
The "hacking" is entirely different in that case. It's not like the person gets sudo power over your computer, it's that your car goes through a stop sign without stopping. It's easy to find the problem because you know exactly what the car should have seen, and fix that thing.
wepple 9y ago
Two points: - you suggest that software is easy and logical, in that you can triage, locate, and patch the buffer overflow. But in any sufficiently large codebase, there are simply near-unlimited more bugs to find: hence all the memory mitigation’s that have been implemented. So it’s not a robust comparison from the get go. - there’s constantly increasing research into being able to diagnose & analyze what CNN’s “see”: possibly not to the level and accuracy that you’d expect from lldb, but how long did it take to have really amazing debuggers for binary Applications?
- resource0x 9y ago
  In a normal program, given an exploit, it's easy to understand the idea of attack and specific target of attack. In CNN case, because no one really understands how things work (see https://www.technologyreview.com/s/604087/the-dark-secret-at-the-heart-of-ai/ https://www.technologyreview.com/s/604087/the-dark-secret-at...), how can they understand why things don't work? The whole contraption is a black box.
  - wepple 9y ago
    > The whole contraption is a black box. That’s not true: there are massive amounts we can glean. Perhaps not as easy as reading the disassembly, but not black box. That link is far outdated/superseded by modern research: https://distill.pub/2018/building-blocks/ https://distill.pub/2018/building-blocks/
  - Too 9y ago
    What about the latest Intel bug Meltdown? It's not always easy to understand such attacks. Also those involving rare race conditions or things that only worked because a core part of your code depended on undefined behavior can require a major refactoring of your whole code base in order to mitigate.

hannasanarion 9y ago

The "hacking" is entirely different in that case. It's not like the person gets sudo power over your computer, it's that your car goes through a stop sign without stopping. It's easy to find the problem because you know exactly what the car should have seen, and fix that thing.

wepple 9y ago

Two points: - you suggest that software is easy and logical, in that you can triage, locate, and patch the buffer overflow. But in any sufficiently large codebase, there are simply near-unlimited more bugs to find: hence all the memory mitigation’s that have been implemented. So it’s not a robust comparison from the get go. - there’s constantly increasing research into being able to diagnose & analyze what CNN’s “see”: possibly not to the level and accuracy that you’d expect from lldb, but how long did it take to have really amazing debuggers for binary Applications?

resource0x 9y ago
In a normal program, given an exploit, it's easy to understand the idea of attack and specific target of attack. In CNN case, because no one really understands how things work (see https://www.technologyreview.com/s/604087/the-dark-secret-at-the-heart-of-ai/ https://www.technologyreview.com/s/604087/the-dark-secret-at...), how can they understand why things don't work? The whole contraption is a black box.
- wepple 9y ago
  > The whole contraption is a black box. That’s not true: there are massive amounts we can glean. Perhaps not as easy as reading the disassembly, but not black box. That link is far outdated/superseded by modern research: https://distill.pub/2018/building-blocks/ https://distill.pub/2018/building-blocks/
- Too 9y ago
  What about the latest Intel bug Meltdown? It's not always easy to understand such attacks. Also those involving rare race conditions or things that only worked because a core part of your code depended on undefined behavior can require a major refactoring of your whole code base in order to mitigate.