4 ms·

The problem is that one is explainable and the other is not. "He was near the scene of the crime and has no convincing alibi," is very different from, "The com

The problem is that one is explainable and the other is not. "He was near the scene of the crime and has no convincing alibi," is very different from, "The computer said he's a suspect and we don't know why, but we still want a warrant." People being targeted for being anomalous is bad, but centralizing and scaling it up is worse.

BurningFrog 6y ago
To me "being a suspect" is a long way from a warrant being issued. If cops can get a warrant on mere hunches, then I agree we have real problems. But a separate problem.
chronic3802 6y ago
> The computer said he's a suspect and we don't know why, but we still want a warrant. Mission critical ML systems (G/FB ads, crime forensics, medical decision support, financial algorithms) do not work like this. The designers know exactly which features are causing responses/predictions in the model, their respective perturbation sensitivities, and have clear bounds regarding adversarial inputs/outliers. “Not knowing how the model works” is definitely true for deep learning, though.
- thylacine222 6y ago
  This seems... incorrect. First of all, deep learning absolutely is used in mission critical systems: https://www.techrepublic.com/article/intel-and-ge-healthcares-x-ray-machine-uses-embedded-ai-to-prioritize-scans/ https://www.techrepublic.com/article/intel-and-ge-healthcare... Second, simply because designers use systems that are formally interpretable, that doesn't mean that designers know exactly what causes a given response in the model. Formal interpretability means you know what function the model is approximating -- there's a long way from that to a human-readable interpretation. Finally, even if the designers can tell you what features cause a certain response in the model, they can only tell you that for features that are encoded in the data! There are plenty of features that aren't encoded in your data that can nevertheless affect the outcome of a model!
  - OnlyOneCannolo 6y ago
    I don't think the Intel/GE tool counts as mission critical. There are already radiologists reviewing scans. The tool just flags features they may have missed. Remove the tool, and you're back to business as usual. I don't know much about ML. Are you saying that we don't know how to interpret any ML models at all? Regarding your final point, isn't that true with or without ML? Any mission-critical design process should scrutinize the solution to see if it's complete enough and correct enough.
    - thylacine222 6y ago
      That tool has the potential to systematically reduce the likelihood that a certain disease/certain class of people receives extra scrutiny from a radiologist. That's maybe not mission critical on a small scale, but on a large scale it absolutely is. Deep learning models are black box models, which means they are not formally interpretable. You can still sometimes get interpretations out of these models using various methods, but they are fairly underdeveloped, and without actual theories of neural network behavior I don't see that improving anytime soon. You're right, it's true with or without ML. In fact, it's true with human-run systems, too. Consider a police officer who is more likely to pull people over in a certain neighborhood, and that neighborhood was 95% AfAm. IF they were asked why they pulled over more people in that neighborhood, they could say they were discriminating against the neighborhood, which, in and of itself is not racial discrimination. Of course, further inspection of that police officer's records could show that they are biased towards neighborhoods with a high AfAm population, which would be racial discrimination. The same scenario can easily arise in an ML context, but interrogating a machine is a very different context from interrogating a human. First of all, people believe that computers are innately 'unbiased' because they are computers, so making the case that an algorithm is biased is already more difficult. Second, going back to the point I made before -- interpreting a model is not the same as providing a human-readable explanation. Asking a question about racial bias in a model which doesn't even encode for race (as many do, in an ill-conceived attempt to be 'neutral') requires skilled people to understand how to ask the question and how to interpret the answer. There's no plug and play process that one can follow to "scrutinize the solution to see if it's complete enough and correct enough".
- mikem170 6y ago
  I'd like to throw a thought out there: It is a problem if the government uses algorithms or ML systems that are secret and/or cannot be independently vetted by the citizenry. Judges cannot just sign off on warrants and sentencing just because "the computer says so", for all we know the computer may be programmed to say "poor and non-comformist = guilty" or some other nonsense, and/or reprogrammed after every election. We need to be able to trust this stuff.
  - emteycz 6y ago
    Is it what's happening? I thought "computer says so" triggers an investigation, which in turns produces reasoning for the judge, based on which the judge decides? Is there any source that confirms the belief about judges deciding based on computer output in the past or in the future - relevant cases, law, etc?
    - mikem170 6y ago
      The worst I've heard of along these lines currently in practice is predictive policing [0]: First the Sheriff’s Office generates lists of people it considers likely to break the law, based on arrest histories, unspecified intelligence and arbitrary decisions by police analysts. Then it sends deputies to find and interrogate anyone whose name appears, often without probable cause, a search warrant or evidence of a specific crime. They swarm homes in the middle of the night, waking families and embarrassing people in front of their neighbors. They write tickets for missing mailbox numbers and overgrown grass, saddling residents with court dates and fines. They come again and again, making arrests for any reason they can. One former deputy described the directive like this: “Make their lives miserable until they move or sue.” It seems that police departments are happy to buy tools for stuff like this, and private companies are happy to make a buck. But are these systems vetted by anyone on behalf of the public? Some like sting rays and breathalyzers have been hidden behind non-disclosure agreements, kept out of open court, etc. Other than the above I've also heard of computer systems in NJ advising judges whether a suspect is a risk of not appearing in court (as they are reforming the bail system). I wondered if the criteria is published, how it reviewed and modified, etc. I'm just concerned about the increasing influence of hidden algorithms on our society, and very concerned in general that the government is going to hook all of its databases together and do more of this, a la Chinese social scores, etc. [0] https://www.techdirt.com/articles/20200907/12212945257/florida-sheriffs-predictive-policing-program-is-protecting-residents-unkempt-lawns-missing-mailbox-numbers.shtml https://www.techdirt.com/articles/20200907/12212945257/flori...
- naewonawonga 6y ago
  >> Mission critical ML systems (G/FB ads, crime forensics, medical decision support, financial algorithms) do not work like this. Are you speaking from experience or from intuition?
rbanffy 6y ago
> "The computer said he's a suspect and we don't know why, but we still want a warrant." We are not talking about black box models here. "Phones that pinged near the Capitol" is a very specific query. They want to identify those who trespassed and those who aided the trespassers. I believe it's pretty fair.
mannykannot 6y ago
I doubt that is something to be much concerned about, as I doubt there would be much use of anything that produces lists of suspects without explanation. What would one do with such a list? It would be like getting a bunch of anonymous tips all saying that a different person did the deed, without any clues to follow up on. Getting a list of suspects is rarely a problem for law enforcement; the difficulty is in winnowing it down to the actual culprits. When a body is found, for example, family and acquaintances are all initially suspects, and experience has shown that summarily dismissing any of them, merely on intuitive grounds, will eliminate some fraction of actual culprits. If a system did start suspecting the actual culprits with a significantly higher success rate than people achieve, there would be much reason to reverse-engineer the process in order to figure out how this was accomplished, as doing so would provide clues (and, ultimately, evidence) that otherwise could only be found by an independent process. This assumes that due process exists, such that unsupported accusations are not taken as evidence, but if due process has been abandoned, we would have a much greater problem than that posited here.
- mbrumlow 6y ago
  > produces lists of suspects without explanation. No. They will come up with some fancy sounding term to describe these situations. Some experts will agree this is a sound method and it will be used to get warrants. They will just put the cause as "suspect by unattended abnormalities" (aka phone not used during crime) Everybody will smile. Warrants will be made. Life's turned up side down. And likely false arrest and convictions. Even if we program rules into th3 AI to avoid this there is a real chance that the AI could work around and add people to thr list that it suspects is involved in one crime but could not list thr suspect do to rules, but a new crime opened a way for it to allow a suspect of crime one to br listed on crime 2.
2OEH8eoCRo0 6y ago
It means the person is a suspect. It means we might have to ask them a few questions. Doesn't mean they did anything.

BurningFrog 6y ago

To me "being a suspect" is a long way from a warrant being issued. If cops can get a warrant on mere hunches, then I agree we have real problems. But a separate problem.

chronic3802 6y ago

> The computer said he's a suspect and we don't know why, but we still want a warrant. Mission critical ML systems (G/FB ads, crime forensics, medical decision support, financial algorithms) do not work like this. The designers know exactly which features are causing responses/predictions in the model, their respective perturbation sensitivities, and have clear bounds regarding adversarial inputs/outliers. “Not knowing how the model works” is definitely true for deep learning, though.

thylacine222 6y ago
This seems... incorrect. First of all, deep learning absolutely is used in mission critical systems: https://www.techrepublic.com/article/intel-and-ge-healthcares-x-ray-machine-uses-embedded-ai-to-prioritize-scans/ https://www.techrepublic.com/article/intel-and-ge-healthcare... Second, simply because designers use systems that are formally interpretable, that doesn't mean that designers know exactly what causes a given response in the model. Formal interpretability means you know what function the model is approximating -- there's a long way from that to a human-readable interpretation. Finally, even if the designers can tell you what features cause a certain response in the model, they can only tell you that for features that are encoded in the data! There are plenty of features that aren't encoded in your data that can nevertheless affect the outcome of a model!
- OnlyOneCannolo 6y ago
  I don't think the Intel/GE tool counts as mission critical. There are already radiologists reviewing scans. The tool just flags features they may have missed. Remove the tool, and you're back to business as usual. I don't know much about ML. Are you saying that we don't know how to interpret any ML models at all? Regarding your final point, isn't that true with or without ML? Any mission-critical design process should scrutinize the solution to see if it's complete enough and correct enough.
  - thylacine222 6y ago
    That tool has the potential to systematically reduce the likelihood that a certain disease/certain class of people receives extra scrutiny from a radiologist. That's maybe not mission critical on a small scale, but on a large scale it absolutely is. Deep learning models are black box models, which means they are not formally interpretable. You can still sometimes get interpretations out of these models using various methods, but they are fairly underdeveloped, and without actual theories of neural network behavior I don't see that improving anytime soon. You're right, it's true with or without ML. In fact, it's true with human-run systems, too. Consider a police officer who is more likely to pull people over in a certain neighborhood, and that neighborhood was 95% AfAm. IF they were asked why they pulled over more people in that neighborhood, they could say they were discriminating against the neighborhood, which, in and of itself is not racial discrimination. Of course, further inspection of that police officer's records could show that they are biased towards neighborhoods with a high AfAm population, which would be racial discrimination. The same scenario can easily arise in an ML context, but interrogating a machine is a very different context from interrogating a human. First of all, people believe that computers are innately 'unbiased' because they are computers, so making the case that an algorithm is biased is already more difficult. Second, going back to the point I made before -- interpreting a model is not the same as providing a human-readable explanation. Asking a question about racial bias in a model which doesn't even encode for race (as many do, in an ill-conceived attempt to be 'neutral') requires skilled people to understand how to ask the question and how to interpret the answer. There's no plug and play process that one can follow to "scrutinize the solution to see if it's complete enough and correct enough".
mikem170 6y ago
I'd like to throw a thought out there: It is a problem if the government uses algorithms or ML systems that are secret and/or cannot be independently vetted by the citizenry. Judges cannot just sign off on warrants and sentencing just because "the computer says so", for all we know the computer may be programmed to say "poor and non-comformist = guilty" or some other nonsense, and/or reprogrammed after every election. We need to be able to trust this stuff.
- emteycz 6y ago
  Is it what's happening? I thought "computer says so" triggers an investigation, which in turns produces reasoning for the judge, based on which the judge decides? Is there any source that confirms the belief about judges deciding based on computer output in the past or in the future - relevant cases, law, etc?
  - mikem170 6y ago
    The worst I've heard of along these lines currently in practice is predictive policing [0]: First the Sheriff’s Office generates lists of people it considers likely to break the law, based on arrest histories, unspecified intelligence and arbitrary decisions by police analysts. Then it sends deputies to find and interrogate anyone whose name appears, often without probable cause, a search warrant or evidence of a specific crime. They swarm homes in the middle of the night, waking families and embarrassing people in front of their neighbors. They write tickets for missing mailbox numbers and overgrown grass, saddling residents with court dates and fines. They come again and again, making arrests for any reason they can. One former deputy described the directive like this: “Make their lives miserable until they move or sue.” It seems that police departments are happy to buy tools for stuff like this, and private companies are happy to make a buck. But are these systems vetted by anyone on behalf of the public? Some like sting rays and breathalyzers have been hidden behind non-disclosure agreements, kept out of open court, etc. Other than the above I've also heard of computer systems in NJ advising judges whether a suspect is a risk of not appearing in court (as they are reforming the bail system). I wondered if the criteria is published, how it reviewed and modified, etc. I'm just concerned about the increasing influence of hidden algorithms on our society, and very concerned in general that the government is going to hook all of its databases together and do more of this, a la Chinese social scores, etc. [0] https://www.techdirt.com/articles/20200907/12212945257/florida-sheriffs-predictive-policing-program-is-protecting-residents-unkempt-lawns-missing-mailbox-numbers.shtml https://www.techdirt.com/articles/20200907/12212945257/flori...
naewonawonga 6y ago
>> Mission critical ML systems (G/FB ads, crime forensics, medical decision support, financial algorithms) do not work like this. Are you speaking from experience or from intuition?

rbanffy 6y ago

> "The computer said he's a suspect and we don't know why, but we still want a warrant." We are not talking about black box models here. "Phones that pinged near the Capitol" is a very specific query. They want to identify those who trespassed and those who aided the trespassers. I believe it's pretty fair.

mannykannot 6y ago

I doubt that is something to be much concerned about, as I doubt there would be much use of anything that produces lists of suspects without explanation. What would one do with such a list? It would be like getting a bunch of anonymous tips all saying that a different person did the deed, without any clues to follow up on. Getting a list of suspects is rarely a problem for law enforcement; the difficulty is in winnowing it down to the actual culprits. When a body is found, for example, family and acquaintances are all initially suspects, and experience has shown that summarily dismissing any of them, merely on intuitive grounds, will eliminate some fraction of actual culprits. If a system did start suspecting the actual culprits with a significantly higher success rate than people achieve, there would be much reason to reverse-engineer the process in order to figure out how this was accomplished, as doing so would provide clues (and, ultimately, evidence) that otherwise could only be found by an independent process. This assumes that due process exists, such that unsupported accusations are not taken as evidence, but if due process has been abandoned, we would have a much greater problem than that posited here.

mbrumlow 6y ago
> produces lists of suspects without explanation. No. They will come up with some fancy sounding term to describe these situations. Some experts will agree this is a sound method and it will be used to get warrants. They will just put the cause as "suspect by unattended abnormalities" (aka phone not used during crime) Everybody will smile. Warrants will be made. Life's turned up side down. And likely false arrest and convictions. Even if we program rules into th3 AI to avoid this there is a real chance that the AI could work around and add people to thr list that it suspects is involved in one crime but could not list thr suspect do to rules, but a new crime opened a way for it to allow a suspect of crime one to br listed on crime 2.

2OEH8eoCRo0 6y ago

It means the person is a suspect. It means we might have to ask them a few questions. Doesn't mean they did anything.