3 ms·

Bad stats make me angry. It's a rolling theme on my blog, though usually I focus my ire on people trying to make people buy more stuff on the internet. https:/

Bad stats make me angry. It's a rolling theme on my blog, though usually I focus my ire on people trying to make people buy more stuff on the internet. https://www.chrisstucchio.com/blog/2015/no_free_samples.html https://www.chrisstucchio.com/blog/2015/no_free_samples.html https://www.chrisstucchio.com/blog/2015/ab_testing_segments_and_goals.html https://www.chrisstucchio.com/blog/2015/ab_testing_segments_... https://www.chrisstucchio.com/pubs/slides/crunch_conf_2015/slides.html https://www.chrisstucchio.com/pubs/slides/crunch_conf_2015/s... Now, most of the folks pushing bad stats are simply not understanding the subtleties of a complex field. PP, in contrast, did understand and ignored the results they got. Yep, that's dishonesty in my book. What would you call it? I have no idea what you are talking about with "machine learning" - it's a synonym for "statistical analysis", leaning a bit in practice towards external validation of models without regard to internal validity. Which is, incidentally, what PP was trying to do. If you want to draw some fine line where a random forest is machine learning but a Cox model isn't, be my guest. I'm sure that's a great reason to ignore PP's own analysis. And you can't stand that they went out and talked to people... No, I can't stand that they cherry-picked anecdotes to push a conclusion that's the exact opposite of what their own analysis claims. Now if you want to criticize the analysis, go ahead. But that's not really a good way to defend PP - after all, if the analysis is bunk, doesn't that discredit the whole article? um, there's an actual Constitutional process to adjust things in our judicial system that shouldn't just be overridden because statistics If you want to make that argument, do it. But that's not the argument PP made. As for my own biases, I'm pretty open and honest in my beliefs in the power of technology. I also back up my claims with math that actually agrees with me, unlike PP. Do you have any objection to what I wrote beyond the fact that you dislike the tone, and might define "machine learning" slightly differently than me?

danso 10y ago
> I have no idea what you are talking about with "machine learning" You have no idea what machine learning is? Then why do you use that phrase. My interpretation is that it is differentiated from statistical analysis in that there's a system that adapts its heuristics to trained and incoming data. For someone to call COMPAS "machine learning" is to have a faith that this system would be adequately tuned and adapted to changing data. For example, if the factors that predicted higher recidivism for blacks were first devised in an era where low-level drug possession was a criminal offense...and now that era no longer exists, and there's no mechanism to adjust for that...I would say that would be a big fucking problem for every criminal who is processed under COMPAS today (regardless of the argument that drug possession crimes were more punitive towards minorities). But we don't know because COMPAS is secret, which was the main thrust of this article. And it seems to perpetuate the biases of the "non-machine" justice system, except in an untransparent way. I said: > um, there's an actual Constitutional process to adjust things in our judicial system that shouldn't just be overridden because statistics you said: > If you want to make that argument, do it. But that's not the argument PP made. Did we even read the same article? The entire fucking article is about whether the justice system is unduly influenced by a proprietary algorithm that has been sold to the system (and paid for by our tax dollars) but is not as smart or objective as it is sold to be. > Scores like this — known as risk assessments — are increasingly common in courtrooms across the nation. They are used to inform decisions about who can be set free at every stage of the criminal justice system, from assigning bond amounts — as is the case in Fort Lauderdale — to even more fundamental decisions about defendants’ freedom. In Arizona, Colorado, Delaware, Kentucky, Louisiana, Oklahoma, Virginia, Washington and Wisconsin, the results of such assessments are given to judges during criminal sentencing. I agree with you that 0.05 < 0.057. What we disagree on is how much of ProPublica's article is based on that, and whether that difference is enough to call them bald-faced liars, given that they acknowledge the limitations of their data collection and show their work and their data. They're attempting to reverse-engineer a black box that's being paid for by taxpayer dollars. They shouldn't have had to provide an anecdote-laced story with statistical analysis to pressure the judicial system for more transparency, but that's what they've chosen to do.
- yummyfajitas 10y ago
  My interpretation is that it is differentiated from statistical analysis in that there's a system that adapts its heuristics to trained and incoming data. The Cox model fits that definition. Perhaps you mean ML is synonymous with reinforcement learning, which is I think the closest thing to "tuned and adapted to changing data"? My interpretation of ML is that it's about evaluating statistical predictors without worrying about their internals. Those are the fundamental advances that have given us all the classical "machine learning" algorithms - random forests, SVMs, neural networks, etc. And the entire point of those techniques is that you can use them to evaluate a secret algorithm. Did we even read the same article? The entire fucking article is about whether the justice system is unduly influenced by a proprietary algorithm that has been sold to the system (and paid for by our tax dollars) but is not as smart or objective as it is sold to be. The entire article claims the algorithm is both inaccurate and racially biased, which their statistical analysis directly contradicts. They never once made the argument that an accurate and unbiased model should not be used.

danso 10y ago

> I have no idea what you are talking about with "machine learning" You have no idea what machine learning is? Then why do you use that phrase. My interpretation is that it is differentiated from statistical analysis in that there's a system that adapts its heuristics to trained and incoming data. For someone to call COMPAS "machine learning" is to have a faith that this system would be adequately tuned and adapted to changing data. For example, if the factors that predicted higher recidivism for blacks were first devised in an era where low-level drug possession was a criminal offense...and now that era no longer exists, and there's no mechanism to adjust for that...I would say that would be a big fucking problem for every criminal who is processed under COMPAS today (regardless of the argument that drug possession crimes were more punitive towards minorities). But we don't know because COMPAS is secret, which was the main thrust of this article. And it seems to perpetuate the biases of the "non-machine" justice system, except in an untransparent way. I said: > um, there's an actual Constitutional process to adjust things in our judicial system that shouldn't just be overridden because statistics you said: > If you want to make that argument, do it. But that's not the argument PP made. Did we even read the same article? The entire fucking article is about whether the justice system is unduly influenced by a proprietary algorithm that has been sold to the system (and paid for by our tax dollars) but is not as smart or objective as it is sold to be. > Scores like this — known as risk assessments — are increasingly common in courtrooms across the nation. They are used to inform decisions about who can be set free at every stage of the criminal justice system, from assigning bond amounts — as is the case in Fort Lauderdale — to even more fundamental decisions about defendants’ freedom. In Arizona, Colorado, Delaware, Kentucky, Louisiana, Oklahoma, Virginia, Washington and Wisconsin, the results of such assessments are given to judges during criminal sentencing. I agree with you that 0.05 < 0.057. What we disagree on is how much of ProPublica's article is based on that, and whether that difference is enough to call them bald-faced liars, given that they acknowledge the limitations of their data collection and show their work and their data. They're attempting to reverse-engineer a black box that's being paid for by taxpayer dollars. They shouldn't have had to provide an anecdote-laced story with statistical analysis to pressure the judicial system for more transparency, but that's what they've chosen to do.

yummyfajitas 10y ago
My interpretation is that it is differentiated from statistical analysis in that there's a system that adapts its heuristics to trained and incoming data. The Cox model fits that definition. Perhaps you mean ML is synonymous with reinforcement learning, which is I think the closest thing to "tuned and adapted to changing data"? My interpretation of ML is that it's about evaluating statistical predictors without worrying about their internals. Those are the fundamental advances that have given us all the classical "machine learning" algorithms - random forests, SVMs, neural networks, etc. And the entire point of those techniques is that you can use them to evaluate a secret algorithm. Did we even read the same article? The entire fucking article is about whether the justice system is unduly influenced by a proprietary algorithm that has been sold to the system (and paid for by our tax dollars) but is not as smart or objective as it is sold to be. The entire article claims the algorithm is both inaccurate and racially biased, which their statistical analysis directly contradicts. They never once made the argument that an accurate and unbiased model should not be used.