4 ms·

Interesting article, but this stood out to me. It frustrates me to see people in the smash community treat measures like elo as "the truth" because they "don't

by jpk 9y ago

Interesting article, but this stood out to me. It frustrates me to see people in the smash community treat measures like elo as "the truth" because they "don't have any human input". This simply factually incorrect - these so-called objective measures have as much human input as anything else, codified into the constants and design choices of their algorithms. Designing these things is as much an art as it is a science, and the choice on how to weigh placements, upsets, losses, consistency, peaks, and the like are all just that - choices, made by a human sitting in a chair with Sublime Text 3 open. I feel like this is applicable nigh everywhere. From social media timeline sorting, to industrial processes, to Melee rankings. Using an algorithm doesn't eliminate the human element from a system, it only abstracts it away.

meesles 9y ago
I'd take it even one step further and apply this to literally anything on the internet! Whether it's news (doesn't even have to be true these days), data (omitted/modified or not), opinions (externally-motivated), research (funded by god-knows-who), literally ANYTHING you view is in some way processed, designed, delivered, or created by a human being. And even when AI becomes practical, no more than 1000 people can realistically be involved in its design and implementation. Our supposed 'objective machines' will in fact be designed to the ideals of those designing them: a generally non-diverse group of people. Food for thought!
SubiculumCode 9y ago
In modeling memory discrimination in psychology experiments, people often recommend d-prime from Signal Detection Theory (https://en.wikipedia.org/wiki/Sensitivity_index https://en.wikipedia.org/wiki/Sensitivity_index). Other recommend simple "theory-free" discrimination scores (Hits-False Alarms). This is frustrating because all measurement designs carry theoretical distributional and metric assumptions, just because one is explicit (as in signal detection theory: d' = Z(hit rate) − Z(false alarm rate), where function Z(p), p ∈ [0,1], is the inverse of the cumulative distribution function of the Gaussian distribution.) does not mean the simple (hit rate minus false alarm rate) is theory free an unadulterated. The theoretical assumptions are just different.
rspeer 9y ago
There is a large contingent of radical empiricists in machine learning who assume "big data + automation = truth", especially on HN, and this is a message they need to hear more of. People have been advocating radical empiricism in some increasingly uncomfortable contexts recently, and I hope it's just that it's the only thing they were taught and the only way they know how to think about their craft. The alternative is that an increasing number of people really do want machines to triumph over human judgment and morality.
- digi_owl 9y ago
  Gets me thinking of an Adam Curtis' documentary series: https://en.wikipedia.org/wiki/All_Watched_Over_by_Machines_of_Loving_Grace_%28TV_series%29 https://en.wikipedia.org/wiki/All_Watched_Over_by_Machines_o... Not exactly sure how relevant it is though.
- bo1024 9y ago
  This might be a case where classical econ ("social choice") can give some helpful perspective. Arrow's impossibility theorem is the most famous impossibility result in this area; it says that no algorithm can take in a set of rankings (e.g. match outcomes) and produce an aggregate ranking in a way that satisfies a small set of fairness criteria. This is classically interpreted as saying that any aggregation method must be "unfair" in one way or another.
  - YokoZar 9y ago
    Arrow's theorem is mathematics, but even citing it is greatly misleading. All Arrow proved was that, sometimes, voting becomes rock-paper-scissors between the three most popular candidates. That isn't "unfair". It especially doesn't mean that all voting systems are equally bad, and yet the most common reason for people to cite Arrow's theorem is to try and dispute the idea that we can do better.
    - thaumasiotes 9y ago
      Unlike your parent comment, your comment is total nonsense. Arrow proved that, given a set of ordinal preference rankings held by several individuals, the concept of an aggregate preference ranking describing the overall "will of society" is not well defined; subject to four or five unobjectionable constraints, no function determining such an overall ranking exists. Voting systems which obey the assumptions of the theorem frequently break down in ways that cannot be described as "rock-paper-scissors between the three most popular candidates", as when strategic voting caused the papal conclave of 1334 to unanimously elect the least popular candidate to the papacy. Were you perhaps thinking of Condorcet's work when you referred to rock-paper-scissors? He, not Arrow, famously wrote about how voter preferences were not necessarily transitive.
Blaaguuu 9y ago
I'm in the middle of reading a pretty interesting book, where that is one of the core arguments - Weapons of Math Destruction, by Cathy O'Neil - I recommend checking it out if you are curious to learn more about the distinction that you made, and more of how we tend to abuse math through algorithms that are in some way designed by humans.
basseq 9y ago
It's "a truth", which is probably as good as you can get, given that "the truth" is always a subjective thing. The benefit to algorithm isn't that it's infallible (it may well be), but rather than it's consistent. It's accurate, even if not correct. Considering how much of human judgement is inconsistent, there's value in quantifying it in a standard way. Reasonable minds can argue about whether than quantification is correct or fair.
- gowld 9y ago
  Why is accurate (by which you mean 'consistent') better than correct?
  - a_t48 9y ago
    Isn't accuracy an aspect of correctness? Ie: can you be correct without being accuracy?
  - da_chicken 9y ago
    Primarily because "correctness" is likely to be subjective or difficult to define, while "accuracy" is not. That said, the terms more often used here are "precise, but not accurate" (http://blog.minitab.com/blog/real-world-quality-improvement/accuracy-vs-precision-whats-the-difference http://blog.minitab.com/blog/real-world-quality-improvement/...) but note that this uses a different meaning for accuracy than parent used. For example, say you use 22/7 to derive the value of pi. You can easily calculate it to 60 or more decimal places. You'll be very precise. However, you won't be accurate because your methodology of using 22/7 is flawed. It's also very easy to create more precision: just keep calculating. On the other hand, it can be very difficult to create more accuracy: How do we even measure pi to be able to confirm the ratio is correct?
  - basseq 9y ago
    It's not. But many "legacy" systems—using the term broadly to mean people and process—are inconsistent and not any more correct as a whole. A standardized system is easier to monitor and easier to correct, so I see this as an improvement overall for that reason.
jdtang13 9y ago
This is true for many statistical analyses as well. The worst is when people smuggle in their non-quantifiable base assumptions and pretend like the resulting inference is some objectively true reality.

by jpk 9y ago

meesles 9y ago

I'd take it even one step further and apply this to literally anything on the internet! Whether it's news (doesn't even have to be true these days), data (omitted/modified or not), opinions (externally-motivated), research (funded by god-knows-who), literally ANYTHING you view is in some way processed, designed, delivered, or created by a human being. And even when AI becomes practical, no more than 1000 people can realistically be involved in its design and implementation. Our supposed 'objective machines' will in fact be designed to the ideals of those designing them: a generally non-diverse group of people. Food for thought!

SubiculumCode 9y ago

In modeling memory discrimination in psychology experiments, people often recommend d-prime from Signal Detection Theory (https://en.wikipedia.org/wiki/Sensitivity_index https://en.wikipedia.org/wiki/Sensitivity_index). Other recommend simple "theory-free" discrimination scores (Hits-False Alarms). This is frustrating because all measurement designs carry theoretical distributional and metric assumptions, just because one is explicit (as in signal detection theory: d' = Z(hit rate) − Z(false alarm rate), where function Z(p), p ∈ [0,1], is the inverse of the cumulative distribution function of the Gaussian distribution.) does not mean the simple (hit rate minus false alarm rate) is theory free an unadulterated. The theoretical assumptions are just different.

rspeer 9y ago

There is a large contingent of radical empiricists in machine learning who assume "big data + automation = truth", especially on HN, and this is a message they need to hear more of. People have been advocating radical empiricism in some increasingly uncomfortable contexts recently, and I hope it's just that it's the only thing they were taught and the only way they know how to think about their craft. The alternative is that an increasing number of people really do want machines to triumph over human judgment and morality.

digi_owl 9y ago
Gets me thinking of an Adam Curtis' documentary series: https://en.wikipedia.org/wiki/All_Watched_Over_by_Machines_of_Loving_Grace_%28TV_series%29 https://en.wikipedia.org/wiki/All_Watched_Over_by_Machines_o... Not exactly sure how relevant it is though.
bo1024 9y ago
This might be a case where classical econ ("social choice") can give some helpful perspective. Arrow's impossibility theorem is the most famous impossibility result in this area; it says that no algorithm can take in a set of rankings (e.g. match outcomes) and produce an aggregate ranking in a way that satisfies a small set of fairness criteria. This is classically interpreted as saying that any aggregation method must be "unfair" in one way or another.
- YokoZar 9y ago
  Arrow's theorem is mathematics, but even citing it is greatly misleading. All Arrow proved was that, sometimes, voting becomes rock-paper-scissors between the three most popular candidates. That isn't "unfair". It especially doesn't mean that all voting systems are equally bad, and yet the most common reason for people to cite Arrow's theorem is to try and dispute the idea that we can do better.
  - thaumasiotes 9y ago
    Unlike your parent comment, your comment is total nonsense. Arrow proved that, given a set of ordinal preference rankings held by several individuals, the concept of an aggregate preference ranking describing the overall "will of society" is not well defined; subject to four or five unobjectionable constraints, no function determining such an overall ranking exists. Voting systems which obey the assumptions of the theorem frequently break down in ways that cannot be described as "rock-paper-scissors between the three most popular candidates", as when strategic voting caused the papal conclave of 1334 to unanimously elect the least popular candidate to the papacy. Were you perhaps thinking of Condorcet's work when you referred to rock-paper-scissors? He, not Arrow, famously wrote about how voter preferences were not necessarily transitive.

Blaaguuu 9y ago

I'm in the middle of reading a pretty interesting book, where that is one of the core arguments - Weapons of Math Destruction, by Cathy O'Neil - I recommend checking it out if you are curious to learn more about the distinction that you made, and more of how we tend to abuse math through algorithms that are in some way designed by humans.

basseq 9y ago

It's "a truth", which is probably as good as you can get, given that "the truth" is always a subjective thing. The benefit to algorithm isn't that it's infallible (it may well be), but rather than it's consistent. It's accurate, even if not correct. Considering how much of human judgement is inconsistent, there's value in quantifying it in a standard way. Reasonable minds can argue about whether than quantification is correct or fair.

gowld 9y ago
Why is accurate (by which you mean 'consistent') better than correct?
- a_t48 9y ago
  Isn't accuracy an aspect of correctness? Ie: can you be correct without being accuracy?
- da_chicken 9y ago
  Primarily because "correctness" is likely to be subjective or difficult to define, while "accuracy" is not. That said, the terms more often used here are "precise, but not accurate" (http://blog.minitab.com/blog/real-world-quality-improvement/accuracy-vs-precision-whats-the-difference http://blog.minitab.com/blog/real-world-quality-improvement/...) but note that this uses a different meaning for accuracy than parent used. For example, say you use 22/7 to derive the value of pi. You can easily calculate it to 60 or more decimal places. You'll be very precise. However, you won't be accurate because your methodology of using 22/7 is flawed. It's also very easy to create more precision: just keep calculating. On the other hand, it can be very difficult to create more accuracy: How do we even measure pi to be able to confirm the ratio is correct?
- basseq 9y ago
  It's not. But many "legacy" systems—using the term broadly to mean people and process—are inconsistent and not any more correct as a whole. A standardized system is easier to monitor and easier to correct, so I see this as an improvement overall for that reason.

jdtang13 9y ago

This is true for many statistical analyses as well. The worst is when people smuggle in their non-quantifiable base assumptions and pretend like the resulting inference is some objectively true reality.