4 ms·

> It's not possible to engineer an autonomous system that never fails, but it is possible to engineer one in such a way that it never fails to detect that it h

> It's not possible to engineer an autonomous system that never fails, but it is possible to engineer one in such a way that it never fails to detect that it has failed This seems to me to be a clearly incorrect (and self-contradictory) claim. It entirely depends on your definition of failure. Your analysis seems fine. The big problem is that the "autonomous" driver is using one signal (where are the lines on the edge of the road?) to the near exclusion of all others (is there a large stationary solid object in front of me?) Maybe Tesla should have hired George Hotz (sp?) if only to write a lightweight sanity-check system that could argue with the main system about whether it was working.

lisper 9y ago
> This seems to me to be a clearly incorrect (and self-contradictory) claim. I guess that means I was able to pull the wool over the eyes of all five members of my thesis committee because none of them thought so. > It entirely depends on your definition of failure. Well, yeah, of course. So? There is some subset of states of the world that you label "failure". The guarantee is not that the system never enters one of those states, the guarantee is that if the system enters one of those states it never (well, so extremely rarely that it's "never" for all practical purposes) fails to recognize that it has done so. Why do you find that so implausible?
- Tloewald 9y ago
  Arguing that it's possible to build an autonomous that is sometimes wrong but always knows when it's wrong seems like a stretch to me.
  - mturmon 9y ago
    A system that can sense when it is in a failure mode is analogous to a “detector” in the Neyman-Pearson sense. A detector that says it is OK when it is actually in a failure condition is said to have a missed detection (MD). OTOH, if you say you’ve failed when you haven’t, that’s a false alarm (FA). In general, there is a trade-off between MD and FA. You can drive MD probability to near-zero, but typically at a cost in FA. Again in general, you can’t claim that you can drive MD to zero (other than by also driving FA probability to one) without knowing more about the specifics of the problem. Here, that would be the sensors, etc. In particular, for systems with noisy, continuous data inputs and continuous state spaces -- not even considering real-world messiness -- I would be surprised if you could drive PD to zero without a very high cost in FA probability. As a humbling example, you cannot do this even for detection of whether a signal is present in Gaussian noise. (I.e., PD = 1 is only achievable with PFA = 1!) PD = 1 is a very high standard in any probabilistic system. Discrete-input systems can behave differently.
    - lisper 9y ago
      You've done a great job of describing the problem. It is manifestly possible to drive missed detection rates very close to zero without too many false alarms because humans are capable of driving safely.
      - mannykannot 9y ago
        Ah, yes... you have convinced me - though humans can apply generalized intelligence to the problem, which I imagine is particularly useful in lots of special cases.
        lisper 9y ago
        General intelligence is not needed. Illiterate people can drive. People who can't do math can drive.
        mannykannot 9y ago
        Illiteracy is not incompatible with intelligence.
      - Tloewald 9y ago
        So we can get within epsilon for an undefined delta because humans can do something, although not always. Right, that whole claim about engineering a system that always knows when it’s not working sounds rock solid to me. After all, we can build a human, right?
        lisper 9y ago
        > we can build a human, right? Not yet. But there's no reason to believe we won't be able to eventually.
  - deleted 9y ago
    [deleted]
  - fwip 9y ago
    You can if you dial the specificity down to 0. i.e: Always report "I'm wrong."
- mjrpes 9y ago
  Humans have moments like this too. I remember suddenly driving into dense fog on the freeway and not able to see more than 20 feet in front of me. I had a definite "failure mode" where I slowed down and freaked out because all my previous driving experience was "failing" me on how to avoid a potential accident.
- thaumasiotes 9y ago
  > The guarantee is not that the system never enters one of those states, the guarantee is that if the system enters one of those states it never (well, so extremely rarely that it's "never" for all practical purposes) fails to recognize that it has done so. Why do you find that so implausible? Stated so, that is plausible. However, "it is possible to engineer a system that never fails to detect that it has failed" is not; I claim that any subset of states which is amenable to this level of detectability will exclude some other states that any normal person would also consider to be "failure".
  - lisper 9y ago
    > Stated so, that is plausible. However, "it is possible to engineer a system that never fails to detect that it has failed" is not They seem the same to me. In fact, in 27 years you are the first person to voice this concern. > I claim that any subset of states which is amenable to this level of detectability will exclude some other states that any normal person would also consider to be "failure". Could be, but that would be a publishable result. So... publish it and then we can discuss.
    - gowld 9y ago
      What happens if all your sensors fail, including the one that senses the failure of your sensors? What happens if the power source disconnects?
      - lisper 9y ago
        Having all your sensors fail is actually a very easy case. Imagine if all of your sensors failed: suddenly you could not see, hear, feel, smell, or taste... do you think it would be hard to tell that something was wrong?
        thaumasiotes 9y ago
        Having your sensors fail doesn't mean they're not providing data. It means they're not providing accurate data. In humans, we would call this hallucinating, and humans in fact cannot generally tell that they are hallucinating.

lisper 9y ago

> This seems to me to be a clearly incorrect (and self-contradictory) claim. I guess that means I was able to pull the wool over the eyes of all five members of my thesis committee because none of them thought so. > It entirely depends on your definition of failure. Well, yeah, of course. So? There is some subset of states of the world that you label "failure". The guarantee is not that the system never enters one of those states, the guarantee is that if the system enters one of those states it never (well, so extremely rarely that it's "never" for all practical purposes) fails to recognize that it has done so. Why do you find that so implausible?

Tloewald 9y ago
Arguing that it's possible to build an autonomous that is sometimes wrong but always knows when it's wrong seems like a stretch to me.
- mturmon 9y ago
  A system that can sense when it is in a failure mode is analogous to a “detector” in the Neyman-Pearson sense. A detector that says it is OK when it is actually in a failure condition is said to have a missed detection (MD). OTOH, if you say you’ve failed when you haven’t, that’s a false alarm (FA). In general, there is a trade-off between MD and FA. You can drive MD probability to near-zero, but typically at a cost in FA. Again in general, you can’t claim that you can drive MD to zero (other than by also driving FA probability to one) without knowing more about the specifics of the problem. Here, that would be the sensors, etc. In particular, for systems with noisy, continuous data inputs and continuous state spaces -- not even considering real-world messiness -- I would be surprised if you could drive PD to zero without a very high cost in FA probability. As a humbling example, you cannot do this even for detection of whether a signal is present in Gaussian noise. (I.e., PD = 1 is only achievable with PFA = 1!) PD = 1 is a very high standard in any probabilistic system. Discrete-input systems can behave differently.
  - lisper 9y ago
    You've done a great job of describing the problem. It is manifestly possible to drive missed detection rates very close to zero without too many false alarms because humans are capable of driving safely.
    - mannykannot 9y ago
      Ah, yes... you have convinced me - though humans can apply generalized intelligence to the problem, which I imagine is particularly useful in lots of special cases.
      - lisper 9y ago
        General intelligence is not needed. Illiterate people can drive. People who can't do math can drive.
        mannykannot 9y ago
        Illiteracy is not incompatible with intelligence.
    - Tloewald 9y ago
      So we can get within epsilon for an undefined delta because humans can do something, although not always. Right, that whole claim about engineering a system that always knows when it’s not working sounds rock solid to me. After all, we can build a human, right?
      - lisper 9y ago
        > we can build a human, right? Not yet. But there's no reason to believe we won't be able to eventually.
- deleted 9y ago
  [deleted]
- fwip 9y ago
  You can if you dial the specificity down to 0. i.e: Always report "I'm wrong."
mjrpes 9y ago
Humans have moments like this too. I remember suddenly driving into dense fog on the freeway and not able to see more than 20 feet in front of me. I had a definite "failure mode" where I slowed down and freaked out because all my previous driving experience was "failing" me on how to avoid a potential accident.
thaumasiotes 9y ago
> The guarantee is not that the system never enters one of those states, the guarantee is that if the system enters one of those states it never (well, so extremely rarely that it's "never" for all practical purposes) fails to recognize that it has done so. Why do you find that so implausible? Stated so, that is plausible. However, "it is possible to engineer a system that never fails to detect that it has failed" is not; I claim that any subset of states which is amenable to this level of detectability will exclude some other states that any normal person would also consider to be "failure".
- lisper 9y ago
  > Stated so, that is plausible. However, "it is possible to engineer a system that never fails to detect that it has failed" is not They seem the same to me. In fact, in 27 years you are the first person to voice this concern. > I claim that any subset of states which is amenable to this level of detectability will exclude some other states that any normal person would also consider to be "failure". Could be, but that would be a publishable result. So... publish it and then we can discuss.
  - gowld 9y ago
    What happens if all your sensors fail, including the one that senses the failure of your sensors? What happens if the power source disconnects?
    - lisper 9y ago
      Having all your sensors fail is actually a very easy case. Imagine if all of your sensors failed: suddenly you could not see, hear, feel, smell, or taste... do you think it would be hard to tell that something was wrong?
      - thaumasiotes 9y ago
        Having your sensors fail doesn't mean they're not providing data. It means they're not providing accurate data. In humans, we would call this hallucinating, and humans in fact cannot generally tell that they are hallucinating.