5 ms·

maybe I'm biased because I'm a software engineer, but I really dont see the controversy here. When you realize that an algorithm is just a process to be perfor

maybe I'm biased because I'm a software engineer, but I really dont see the controversy here. When you realize that an algorithm is just a process to be performed, what is new here other than using a computer and a large amount of data? Judges always have been using information to predict the harm that a defendant might do in the future. I dont have the data, but I'm sure that past convictions, antisocial behavior, friends that have pro criminal beliefs, etc do in fact correlate with future criminal behavior. Now I know that correlation is not causation, but still. Is this just because the algorithm is closed source? A judges thought process is closed source also, no? To me this seems like an obvious and good thing. edit: paragraphs

jdmichal 10y ago
I generally agree with you. I see nothing different here than what's done every day by insurance companies looking to assess individual risk. If an algorithm can decide how likely I am to be at fault in a wreck, why can't one decide my chances of recidivism?
- hackuser 10y ago
  Government is held to higher standards than businesses, because government can take your freedom and property. If you don't like your insurance rates, switch to another carrier; the insurance company won't imprison you. As another example, government can't stop you from saying whatever you like (freedom of speech), but your employer can restrict what you say at work.
  - jdmichal 10y ago
    > Government is held to higher standards than businesses, because government can take your freedom and property. So, to be clear, a higher standard means the government is not allowed to use data-driven algorithms to make decisions? Keep in mind that judges already had and continue to have latitude to independently conclude the same thing the algorithm does and act appropriately. Or is your disagreement with the closed nature of the algorithms? Which is an argument, but one that should also be applied to insurance companies, especially those brokering mandatory products like car and health. EDIT: To copy from my other reply: If the problem is in the openness of the decision, then let's open up everything. If the problem is in making data-driven, algorithmic decisions in the first place... I'm not sure that's something I can get behind.
    - hackuser 10y ago
      > a higher standard means the government is not allowed to use data-driven algorithms to make decisions? Keep in mind that judges already had and continue to have latitude to independently conclude the same thing the algorithm does and act appropriately. No, I generally agree with you. However, I'm very concerned that judges will rely too heavily on the computer, either through laziness or through misunderstanding the limitations of an algorithm. > is your disagreement with the closed nature of the algorithms? Mostly, yes. IMO: * A spec, which the code meets, must be published. The spec should be comprehensible to non-technical readers - at least judges and lawyers. (EDIT: The spec governs, not the code that implements it.) * The code also must be open and written to some standards of coding comprehensibility and documentation, so others can verify that it meets the spec. If they just open source some code, it's useless. Even experts such as software developers probably aren't going to quickly grasp someone else's code. > Which is an argument, but one that should also be applied to insurance companies, especially those brokering mandatory products like car and health. I tend to agree, but I think it's still a matter of degree.
- denzil_correa 10y ago
  > If an algorithm can decide how likely I am to be at fault in a wreck, why can't one decide my chances of recidivism? Alternatively, If an algorithm for recidivism is questioned, why should the algorithm for insurance also not be questioned? Two wrongs do not make a right.
  - GFischer 10y ago
    The main difference with this case is that you can shop around for different insurance, and different insurance companies use different algorithms to determine risk. You can, of course, question them. In many cases they're open source, they're usually government-audited, and in some countries, even government mandated. Edit: hackuser beat me to the point - https://news.ycombinator.com/item?id=11957351 https://news.ycombinator.com/item?id=11957351
  - jdmichal 10y ago
    I agree with you completely. If the problem is in the openness of the decision, then let's open up everything. If the problem is in making data-driven, algorithmic decisions in the first place... I'm not sure that's something I can get behind.
    - denzil_correa 10y ago
      Algorithmic decisions aren't made in vacuum. These decisions are made via human inputs which may be biased.
      - jdmichal 10y ago
        That's not really relevant. These decisions are going to be made anyway, and until we have general AI that means human inputs and often something even worse -- human logic. Algorithms remove the latter by standardizing the logic; that's a win in my book. An excerpt from the other thread: > Keep in mind that judges already had and continue to have latitude to independently conclude the same thing the algorithm does and act appropriately.
hackuser 10y ago
> A judges thought process is closed source also, no? No, a judge's thought process is determined by open law, open precedent, and the judge's reasoning is explained openly in the published legal opinion. That reasoning can be challenged in an appeal. Openness in courts is a fundamental of democracy, liberty and human rights. Otherwise, you can't defend yourself and the government doesn't have to explain its actions to the public.
- rayiner 10y ago
  Sentences are not, generally, based on law and precedent. Within a wide range, sentences are heavily dependent on judges' discretion and ad-hoc and unscientific consideration of various factors.
  - nickysielicki 10y ago
    > sentences are heavily dependent on judges' discretion and ad-hoc and unscientific consideration of various factors. Sure, but they are put out in the open, and the judge does justify it in some written account, so that it can be referenced when challenged in an appeals court. What answer should be given to someone who is given a sentence based on this algorithm and challenges it? "Well, the algorithm said so, and studies say that it's pretty accurate, and we don't know how it came to this particular rating of you, so you're shit out of luck?"
    - ahartman00 10y ago
      And the studies are closed as well. edit: since i cant reply to hackuser below, here is a quote from the article: "Company officials say the algorithm’s results are backed by research, but they are tight-lipped about its details. "
      - hackuser 10y ago
        Interesting. Does it say that in the article? EDIT: ahartman00: I'm not sure why you don't see the reply link, but another option is the click the time ('x minutes/hours ago'), and you can reply there. Regarding that quote, I thought it meant they were tight-lipped about the algorithm, not the research, but I see that it's ambiguous.
GFK_of_xmaspast 10y ago
Seriously? http://www.nytimes.com/2015/07/10/upshot/when-algorithms-discriminate.html?_r=0 http://www.nytimes.com/2015/07/10/upshot/when-algorithms-dis... http://spectrum.ieee.org/tech-talk/computing/software/computer-scientists-find-bias-in-algorithms http://spectrum.ieee.org/tech-talk/computing/software/comput... https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing https://www.propublica.org/article/machine-bias-risk-assessm...
- ahartman00 10y ago
  Ok, good points. However they did mention that the sentencing algorithm does not take race into account. It does take gender into account, so :(
  - vkou 10y ago
    In some areas, if the algorithm takes postal code into account, it can, with high probability use it as a proxy for race.
rayiner 10y ago
Consider why people get worked up over civil asset forfeiture of people with large amounts of cash on their person. Statistically, it's highly probable that money is the product of illegal activity--i.e. if you find 10 people carrying 10,000 in their car, only one or two probably obtained that money legitimately. Yet, it strikes people as "unfair" to play the statistics like that, because peoples' sense of justice requires some sort of individualized due process. That is why, for example, you generally can't introduce evidence of past crimes in order to prove guilt for a different crime.
- hackuser 10y ago
  > Consider why people get worked up over civil asset forfeiture of people with large amounts of cash on their person. Statistically, it's highly probable that money is the product of illegal activity I haven't seen any stats about the assets seized; do you know of any?
  - rayiner 10y ago
    80-85% of seized assets go uncontested: http://www.fed-soc.org/publications/detail/forfeiture-is-reasonable-and-it-works http://www.fed-soc.org/publications/detail/forfeiture-is-rea.... I.e. nobody even files a claim to get back their thousands of dollars in cash. You can use your common sense to figure why that's the case. I don't think we should have civil asset forfeiture. But I think it's improbable that more than a small minority of those assets are legitimate (especially in the $10,000 in the glove box scenarios).
    - ahartman00 10y ago
      "You can use your common sense to figure why that's the case" Im not sure about this: "this exploitative practice still stacks the deck against property owners, essentially requiring them to prove their innocence to gain back their belongings. For people whose property has been seized through civil asset forfeiture, legally regaining such property is notoriously difficult and expensive, with costs sometimes exceeding the value of the property" - http://www.aclu-tn.org/civil-asset-forfeiture.html http://www.aclu-tn.org/civil-asset-forfeiture.html
denzil_correa 10y ago
> When you realize that an algorithm is just a process to be performed, what is new here other than using a computer and a large amount of data? This is precisely the problem. If the algorithm is dependent on a certain process, this process should be vetted for potential biases - big data or not.

jdmichal 10y ago

I generally agree with you. I see nothing different here than what's done every day by insurance companies looking to assess individual risk. If an algorithm can decide how likely I am to be at fault in a wreck, why can't one decide my chances of recidivism?

hackuser 10y ago
Government is held to higher standards than businesses, because government can take your freedom and property. If you don't like your insurance rates, switch to another carrier; the insurance company won't imprison you. As another example, government can't stop you from saying whatever you like (freedom of speech), but your employer can restrict what you say at work.
- jdmichal 10y ago
  > Government is held to higher standards than businesses, because government can take your freedom and property. So, to be clear, a higher standard means the government is not allowed to use data-driven algorithms to make decisions? Keep in mind that judges already had and continue to have latitude to independently conclude the same thing the algorithm does and act appropriately. Or is your disagreement with the closed nature of the algorithms? Which is an argument, but one that should also be applied to insurance companies, especially those brokering mandatory products like car and health. EDIT: To copy from my other reply: If the problem is in the openness of the decision, then let's open up everything. If the problem is in making data-driven, algorithmic decisions in the first place... I'm not sure that's something I can get behind.
  - hackuser 10y ago
    > a higher standard means the government is not allowed to use data-driven algorithms to make decisions? Keep in mind that judges already had and continue to have latitude to independently conclude the same thing the algorithm does and act appropriately. No, I generally agree with you. However, I'm very concerned that judges will rely too heavily on the computer, either through laziness or through misunderstanding the limitations of an algorithm. > is your disagreement with the closed nature of the algorithms? Mostly, yes. IMO: * A spec, which the code meets, must be published. The spec should be comprehensible to non-technical readers - at least judges and lawyers. (EDIT: The spec governs, not the code that implements it.) * The code also must be open and written to some standards of coding comprehensibility and documentation, so others can verify that it meets the spec. If they just open source some code, it's useless. Even experts such as software developers probably aren't going to quickly grasp someone else's code. > Which is an argument, but one that should also be applied to insurance companies, especially those brokering mandatory products like car and health. I tend to agree, but I think it's still a matter of degree.
denzil_correa 10y ago
> If an algorithm can decide how likely I am to be at fault in a wreck, why can't one decide my chances of recidivism? Alternatively, If an algorithm for recidivism is questioned, why should the algorithm for insurance also not be questioned? Two wrongs do not make a right.
- GFischer 10y ago
  The main difference with this case is that you can shop around for different insurance, and different insurance companies use different algorithms to determine risk. You can, of course, question them. In many cases they're open source, they're usually government-audited, and in some countries, even government mandated. Edit: hackuser beat me to the point - https://news.ycombinator.com/item?id=11957351 https://news.ycombinator.com/item?id=11957351
- jdmichal 10y ago
  I agree with you completely. If the problem is in the openness of the decision, then let's open up everything. If the problem is in making data-driven, algorithmic decisions in the first place... I'm not sure that's something I can get behind.
  - denzil_correa 10y ago
    Algorithmic decisions aren't made in vacuum. These decisions are made via human inputs which may be biased.
    - jdmichal 10y ago
      That's not really relevant. These decisions are going to be made anyway, and until we have general AI that means human inputs and often something even worse -- human logic. Algorithms remove the latter by standardizing the logic; that's a win in my book. An excerpt from the other thread: > Keep in mind that judges already had and continue to have latitude to independently conclude the same thing the algorithm does and act appropriately.

hackuser 10y ago

> A judges thought process is closed source also, no? No, a judge's thought process is determined by open law, open precedent, and the judge's reasoning is explained openly in the published legal opinion. That reasoning can be challenged in an appeal. Openness in courts is a fundamental of democracy, liberty and human rights. Otherwise, you can't defend yourself and the government doesn't have to explain its actions to the public.

rayiner 10y ago
Sentences are not, generally, based on law and precedent. Within a wide range, sentences are heavily dependent on judges' discretion and ad-hoc and unscientific consideration of various factors.
- nickysielicki 10y ago
  > sentences are heavily dependent on judges' discretion and ad-hoc and unscientific consideration of various factors. Sure, but they are put out in the open, and the judge does justify it in some written account, so that it can be referenced when challenged in an appeals court. What answer should be given to someone who is given a sentence based on this algorithm and challenges it? "Well, the algorithm said so, and studies say that it's pretty accurate, and we don't know how it came to this particular rating of you, so you're shit out of luck?"
  - ahartman00 10y ago
    And the studies are closed as well. edit: since i cant reply to hackuser below, here is a quote from the article: "Company officials say the algorithm’s results are backed by research, but they are tight-lipped about its details. "
    - hackuser 10y ago
      Interesting. Does it say that in the article? EDIT: ahartman00: I'm not sure why you don't see the reply link, but another option is the click the time ('x minutes/hours ago'), and you can reply there. Regarding that quote, I thought it meant they were tight-lipped about the algorithm, not the research, but I see that it's ambiguous.

GFK_of_xmaspast 10y ago

Seriously? http://www.nytimes.com/2015/07/10/upshot/when-algorithms-discriminate.html?_r=0 http://www.nytimes.com/2015/07/10/upshot/when-algorithms-dis... http://spectrum.ieee.org/tech-talk/computing/software/computer-scientists-find-bias-in-algorithms http://spectrum.ieee.org/tech-talk/computing/software/comput... https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing https://www.propublica.org/article/machine-bias-risk-assessm...

ahartman00 10y ago
Ok, good points. However they did mention that the sentencing algorithm does not take race into account. It does take gender into account, so :(
- vkou 10y ago
  In some areas, if the algorithm takes postal code into account, it can, with high probability use it as a proxy for race.

rayiner 10y ago

Consider why people get worked up over civil asset forfeiture of people with large amounts of cash on their person. Statistically, it's highly probable that money is the product of illegal activity--i.e. if you find 10 people carrying 10,000 in their car, only one or two probably obtained that money legitimately. Yet, it strikes people as "unfair" to play the statistics like that, because peoples' sense of justice requires some sort of individualized due process. That is why, for example, you generally can't introduce evidence of past crimes in order to prove guilt for a different crime.

hackuser 10y ago
> Consider why people get worked up over civil asset forfeiture of people with large amounts of cash on their person. Statistically, it's highly probable that money is the product of illegal activity I haven't seen any stats about the assets seized; do you know of any?
- rayiner 10y ago
  80-85% of seized assets go uncontested: http://www.fed-soc.org/publications/detail/forfeiture-is-reasonable-and-it-works http://www.fed-soc.org/publications/detail/forfeiture-is-rea.... I.e. nobody even files a claim to get back their thousands of dollars in cash. You can use your common sense to figure why that's the case. I don't think we should have civil asset forfeiture. But I think it's improbable that more than a small minority of those assets are legitimate (especially in the $10,000 in the glove box scenarios).
  - ahartman00 10y ago
    "You can use your common sense to figure why that's the case" Im not sure about this: "this exploitative practice still stacks the deck against property owners, essentially requiring them to prove their innocence to gain back their belongings. For people whose property has been seized through civil asset forfeiture, legally regaining such property is notoriously difficult and expensive, with costs sometimes exceeding the value of the property" - http://www.aclu-tn.org/civil-asset-forfeiture.html http://www.aclu-tn.org/civil-asset-forfeiture.html

denzil_correa 10y ago

> When you realize that an algorithm is just a process to be performed, what is new here other than using a computer and a large amount of data? This is precisely the problem. If the algorithm is dependent on a certain process, this process should be vetted for potential biases - big data or not.