4 ms·

Should we set a requirement that AI models used in certain decision making have to be "static" and "deterministic"? Meaning it would be possible to inspect and

Should we set a requirement that AI models used in certain decision making have to be "static" and "deterministic"? Meaning it would be possible to inspect and re-run cases through the specific version of model that was in use at certain point in past? You could take the model that was in use 2024-19-02T10:54:00Z, feed in data and get always the same results. Then tinker with the inputs and observe the results to find out for example potential biases or errors.

nl 3y ago
Why? Surely the key question is "Is it correct?", not if the same words were used?
- RandomLensman 3y ago
  In very few situations you get a "yes" or "no" to that kind of question or you have a lot of things that are correct but not all of equal quality etc.
- bilbo0s 3y ago
  Yep. Far more important to, 1 - get a result consistent with both reasonable diagnosis and standard of care for the condition in question. And, 2 - get the same result given the same input every time. The FDA doesn’t like when you say “amputate the left leg” one day, and then say “amputate the right leg” the next day. Particularly when that happens after you’ve already amputated the left leg and there was nothing wrong with it. Believe it or not, that amputation scenario has happened before. It wasn’t with AI software, but rather imaging software that hosed an orthonormal conversion on a DICOM dataset. But the same rules should obviously apply to AI.
  - RandomLensman 3y ago
    This sounds a bit like using a standard manual (not the wrong leg part, though).
    - bilbo0s 3y ago
      Well, they don’t generally think it’s necessary to put “don’t amputate the wrong leg” in the books in med school, but yes, if it helps to think of it that way, you can.
  - rickdeckard 3y ago
    In your leg-example your condition #2 was not met, let's imagine that still happens despite using "AI magic": At the point when "amputate the right leg" was found to be the correct diagnosis, an authority should be able to audit and reconstruct the wrong "amputate left leg" diagnosis, trace the root-cause of this decision and verify the corrective action. For this, it is important to be able to reconstruct the process of any computer-aided recommendation used. A corporate statement like "we don't know why the model recommended the left leg back then, now it correctly recommends the right leg when feeding the same data" should not be sufficient.
_trampeltier 3y ago
Of course it should. In other tech, if somethung is tested with a measurment tool, there are also numbers to calibration certs on your cert. So you can follow back the tools to the national standarts. So yes of course you should know what version checked your data.
rickdeckard 3y ago
Or, in the words of the European Union: "AI systems [need to be] safe, transparent, traceable, non-discriminatory and environmentally friendly." And: "AI systems should be overseen by people, rather than by automation, to prevent harmful outcomes." Source: https://www.europarl.europa.eu/topics/en/article/20230601STO93804/eu-ai-act-first-regulation-on-artificial-intelligence https://www.europarl.europa.eu/topics/en/article/20230601STO...

nl 3y ago

Why? Surely the key question is "Is it correct?", not if the same words were used?

RandomLensman 3y ago
In very few situations you get a "yes" or "no" to that kind of question or you have a lot of things that are correct but not all of equal quality etc.
bilbo0s 3y ago
Yep. Far more important to, 1 - get a result consistent with both reasonable diagnosis and standard of care for the condition in question. And, 2 - get the same result given the same input every time. The FDA doesn’t like when you say “amputate the left leg” one day, and then say “amputate the right leg” the next day. Particularly when that happens after you’ve already amputated the left leg and there was nothing wrong with it. Believe it or not, that amputation scenario has happened before. It wasn’t with AI software, but rather imaging software that hosed an orthonormal conversion on a DICOM dataset. But the same rules should obviously apply to AI.
- RandomLensman 3y ago
  This sounds a bit like using a standard manual (not the wrong leg part, though).
  - bilbo0s 3y ago
    Well, they don’t generally think it’s necessary to put “don’t amputate the wrong leg” in the books in med school, but yes, if it helps to think of it that way, you can.
- rickdeckard 3y ago
  In your leg-example your condition #2 was not met, let's imagine that still happens despite using "AI magic": At the point when "amputate the right leg" was found to be the correct diagnosis, an authority should be able to audit and reconstruct the wrong "amputate left leg" diagnosis, trace the root-cause of this decision and verify the corrective action. For this, it is important to be able to reconstruct the process of any computer-aided recommendation used. A corporate statement like "we don't know why the model recommended the left leg back then, now it correctly recommends the right leg when feeding the same data" should not be sufficient.

_trampeltier 3y ago

Of course it should. In other tech, if somethung is tested with a measurment tool, there are also numbers to calibration certs on your cert. So you can follow back the tools to the national standarts. So yes of course you should know what version checked your data.

rickdeckard 3y ago

Or, in the words of the European Union: "AI systems [need to be] safe, transparent, traceable, non-discriminatory and environmentally friendly." And: "AI systems should be overseen by people, rather than by automation, to prevent harmful outcomes." Source: https://www.europarl.europa.eu/topics/en/article/20230601STO93804/eu-ai-act-first-regulation-on-artificial-intelligence https://www.europarl.europa.eu/topics/en/article/20230601STO...