3 ms·

Isn't it a "high-assurance distributed AI/ML system"? It is interesting that there is indeed a tendency to factorize it. I believe, this tendency arises becau

Isn't it a "high-assurance distributed AI/ML system"? It is interesting that there is indeed a tendency to factorize it. I believe, this tendency arises because there are two groups of people that are available - traditional engineers, that are excellent and experienced, but do not know the language of AI and can't tell an ensemble from an embedding. And AI engineers, who are yet to build their first product.

im_down_w_otp 8y ago
Sort of. Except I don't think there's a such thing as "high-assurance AI/ML". Put simply for the audience here. The goal is to have the whole stack from top to bottom be as predictable and deterministic as possible. There's a layer of it (the AI/ML bits) which is necessarily probabilistic. That's by design. It's how it all works. All the layers below that today are currently terrifyingly probabilistic. It's difficult to advance the capability of the whole domain when the necessarily probabilistic parts are depending on shifting sands as a stable foundation. I agree though that there's a large gap there in the domain expertise of the parties involved, and that's produced significant challenges as we've not yet figured out how to close that gap.
- dchichkov 8y ago
  Are you sure, such thing as "high-assurance AI/ML" doesn't exist? Are you an expert in AI, with many years of experience building products with AI technology?
  - im_down_w_otp 8y ago
    I'm not an expert in AI, but as you can imagine I have to interact with a lot of people who are. What I do know is what it means to be able to do end-to-end verification and it seems fairly antithetical to the basis of machine learning which has an open-ended failure domain. If you could write an AI as a total program, it would cease to be probabilistic. It would become propositional, but would also seem to cease to be writable and computationally tractable. As it would need to encode all possible states of all possible features of all relevant domains and all possible things that can interact with each of those things and what their effects would be. Though, I'm happy to be proven wrong here as it would make me a lot less nervous about the future were there a way to legitimately assure the artificial driver in an AV won't fail in emergent ways, or at all.
    - dchichkov 8y ago
      "What foolishness! What nonsense! .... For as children tremble and fear everything in the blind darkness, so we in the light sometimes fear what is no more to be feared than the things children in the dark hold in terror and imagine will come true. This terror therefore and darkness of mind must be dispelled not by the rays of the sun and glittering shafts of day, but by the aspect and law of nature." LUCRETIUS, De Rerum Natura :)
    - ndpi 8y ago
      I work on the formal verification of AI systems as well, but focusing on model-based reasoning systems rather than the model-free ML systems. What you've described is exactly what terrifies me most about ML in safety critical contexts: it can fail in a large number of ways, and predicting these ahead of time is generally intractable. I like to think of it in terms of quantifiers: for most systems it's fine to say that there exists a set of representative tests that the system passes. For critical systems, you really want "for all" in there instead. Fortunately the model-based systems are much easier to reason about, even if they suffer from difficulty modeling certain cases. For a model-free system I think the starting point would be to precisely state a correctness criterion that could be used as a decision procedure on the output of the model; the combined system would be "verified" to a certain degree. Unfortunately you can't guarantee that it would always be able to yield an answer (it might always give a bad one which was rejected) and, the more fundamental problem, one might reasonably expect that if you can express the correctness property to a sufficient level of detail that you can perform verification on it you might be able to write an easier-to-verify implementation that doesn't rely on ML. This stuff is hard, yo.
- deleted 8y ago
  [deleted]
- IX-103 8y ago
  So, two questions: 1. What parts of AI/ML have to be nondeterministic? Once the model is trained then it should be completely deterministic. There are some training techniques that randomize the order or grouping of data, but they are primarily used because random is almost as good as finding the optimal ordering/grouping and finding the optimal ordering/grouping is hard. Sometimes randomness is used to augment the data to give you more examples to for against (because the model is too complex to train/fit with the data you have it you want to be robust to certain variations in input. If you're talking outputs then you should get the same outputs from a model given the same inputs. There are some really bad models out there that are way over fitted (relevant xkcd - https://m.xkcd.com/2048/ https://m.xkcd.com/2048/ see "house of cards") and some people working on ML have a terrible grasp of statical techniques (just try models until they find one that fits), but even these models are deterministic. What is so bad about nondeterminism anyway? Ethernet used nondeterminism to handle collision resolution. There best prime finding algorithms are nondeterministic. There is nothing wrong with appropriate nondeterminism.

im_down_w_otp 8y ago

Sort of. Except I don't think there's a such thing as "high-assurance AI/ML". Put simply for the audience here. The goal is to have the whole stack from top to bottom be as predictable and deterministic as possible. There's a layer of it (the AI/ML bits) which is necessarily probabilistic. That's by design. It's how it all works. All the layers below that today are currently terrifyingly probabilistic. It's difficult to advance the capability of the whole domain when the necessarily probabilistic parts are depending on shifting sands as a stable foundation. I agree though that there's a large gap there in the domain expertise of the parties involved, and that's produced significant challenges as we've not yet figured out how to close that gap.

dchichkov 8y ago
Are you sure, such thing as "high-assurance AI/ML" doesn't exist? Are you an expert in AI, with many years of experience building products with AI technology?
- im_down_w_otp 8y ago
  I'm not an expert in AI, but as you can imagine I have to interact with a lot of people who are. What I do know is what it means to be able to do end-to-end verification and it seems fairly antithetical to the basis of machine learning which has an open-ended failure domain. If you could write an AI as a total program, it would cease to be probabilistic. It would become propositional, but would also seem to cease to be writable and computationally tractable. As it would need to encode all possible states of all possible features of all relevant domains and all possible things that can interact with each of those things and what their effects would be. Though, I'm happy to be proven wrong here as it would make me a lot less nervous about the future were there a way to legitimately assure the artificial driver in an AV won't fail in emergent ways, or at all.
  - dchichkov 8y ago
    "What foolishness! What nonsense! .... For as children tremble and fear everything in the blind darkness, so we in the light sometimes fear what is no more to be feared than the things children in the dark hold in terror and imagine will come true. This terror therefore and darkness of mind must be dispelled not by the rays of the sun and glittering shafts of day, but by the aspect and law of nature." LUCRETIUS, De Rerum Natura :)
  - ndpi 8y ago
    I work on the formal verification of AI systems as well, but focusing on model-based reasoning systems rather than the model-free ML systems. What you've described is exactly what terrifies me most about ML in safety critical contexts: it can fail in a large number of ways, and predicting these ahead of time is generally intractable. I like to think of it in terms of quantifiers: for most systems it's fine to say that there exists a set of representative tests that the system passes. For critical systems, you really want "for all" in there instead. Fortunately the model-based systems are much easier to reason about, even if they suffer from difficulty modeling certain cases. For a model-free system I think the starting point would be to precisely state a correctness criterion that could be used as a decision procedure on the output of the model; the combined system would be "verified" to a certain degree. Unfortunately you can't guarantee that it would always be able to yield an answer (it might always give a bad one which was rejected) and, the more fundamental problem, one might reasonably expect that if you can express the correctness property to a sufficient level of detail that you can perform verification on it you might be able to write an easier-to-verify implementation that doesn't rely on ML. This stuff is hard, yo.
deleted 8y ago
[deleted]
IX-103 8y ago
So, two questions: 1. What parts of AI/ML have to be nondeterministic? Once the model is trained then it should be completely deterministic. There are some training techniques that randomize the order or grouping of data, but they are primarily used because random is almost as good as finding the optimal ordering/grouping and finding the optimal ordering/grouping is hard. Sometimes randomness is used to augment the data to give you more examples to for against (because the model is too complex to train/fit with the data you have it you want to be robust to certain variations in input. If you're talking outputs then you should get the same outputs from a model given the same inputs. There are some really bad models out there that are way over fitted (relevant xkcd - https://m.xkcd.com/2048/ https://m.xkcd.com/2048/ see "house of cards") and some people working on ML have a terrible grasp of statical techniques (just try models until they find one that fits), but even these models are deterministic. What is so bad about nondeterminism anyway? Ethernet used nondeterminism to handle collision resolution. There best prime finding algorithms are nondeterministic. There is nothing wrong with appropriate nondeterminism.