5 ms·

I’m not so sure it’s a bad idea to listen to first person accounts from underindexed groups. For one, it may help surface new metrics to readjust with; for anot

I’m not so sure it’s a bad idea to listen to first person accounts from underindexed groups. For one, it may help surface new metrics to readjust with; for another, it may still help to get a sampling of “is this helping or hurting in your opinion” and watch how it trends over time.

mlthoughts2018 7y ago
I understand what you mean, but the problem is that first person accounts are so idiosyncratic and situation-specific. It takes a lot of discipline to distill only the most well-supported high level themes from a set of anecdotal qualitative data. The discussion also is at high risk of becoming politicized, where more credence will be given to salient first-person accounts that happen to fit well into some prevalent social narrative of the times. If you can prove that a lot of care was taken to avoid these types of bias in extracting summaries from a collection of qualitative data, then it can be valuable. But often this means you have to be so cautious and err on the side of rejecting all but the most consistent summaries from the data that you run afoul of critics who want the data to be used to support their preferred political narrative. This risk of subverting qualitative data like this into a disingenuous claim that something is data-driven is really high, in addition to all the usual cognitive biases that lead people to read what they want into narrative accounts.
- inimino 7y ago
  > This risk of subverting qualitative data like this into a disingenuous claim that something is data-driven is really high, in addition to all the usual cognitive biases that lead people to read what they want into narrative accounts. We have engaged on related ideas before[1], and I think your position represents something that I find quite common and deeply unsettling about modern tech culture, but which I don't yet feel I can fully characterize. There is something about the need for everything to be "data-driven" and metrics-focused that I think represents an abrogation of moral responsibility to form, apply, and maintain our own value systems. It is easier and we would apparently rather assign this messy work to some external process or algorithm or metric, which can then be argued to be perfectly rational, correct, data-driven, unbiased[2]. To me this is a dangerous fantasy, closely related to scientism or the general failure to accept any reality outside of what can be measured, counted, or quantified. This is a characteristic failure (in my opinion) of people in their twenties and of STEM majors in our times, so it's not surprising that it would be a characteristic failure of tech, where both those categories are over-represented. It also seems correlated with a general ignorance of and arrogance towards the humanities among tech workers. [1]: https://news.ycombinator.com/item?id=21097061 https://news.ycombinator.com/item?id=21097061 [2]: A related symptom of this disease is in hiring, where companies come up with systems that look very logical and bias free on paper, for example having "objective" scoring criteria and technical challenges (which end up being almost random number generators, but with some bias that tends to reinforce existing imbalances in the industry, e.g. towards recent graduates), and equally "objective" ways of combining results from hiring committee members or interviewers that tend to create and enforce conformity and exclude divergent viewpoints, while giving the appearance of being bias-free, data-driven, and rational. In fact it's a great way that entire companies fool themselves and come up with terrible processes, and then they whine about how hard it is to maintain diversity (of gender, age, whatever) in their ranks. The people that design these systems aren't doing this as some kind of great conspiracy; as far as I can see, they're not evil, just blind.
  - perl4ever 7y ago
    "an abrogation of moral responsibility to form, apply, and maintain our own value systems." I think it's a big (and almost certainly invalid) assumption that in the absence of metrics, people "apply and maintain" a consistent value system. It seems to me inconsistent with a belief in the destructive effect of the wrong metrics. If you think that metrics lead to bad results, it's completely inconsistent to suggest that not having metrics is better. When you have metrics, the harm comes from people ignoring what really should be optimized in favor of what is being measured. If you don't have any metrics, then you're only decreasing the pressure to do whatever may be the right thing. It's like, no matter how bad the government is, the only solution is better government, not eliminating it. So, I think your general attitude is ironically capturing the essence of a common intellectual failing of the people you are stereotyping. You know, the glib libertarian tech worker.
    - inimino 7y ago
      > in the absence of metrics, people "apply and maintain" a consistent value system What on earth makes you think I'm making this assumption? I never said anything of the sort. I said that people are abrogating their responsibility to do difficult work (figuring out if the ship is going in the right direction or not) by trying to offload this onto metrics, which are theoretically objective and unbiased. Nowhere did I ever say anything about what people not over-using metrics are doing. > If you don't have any metrics, then you're only decreasing the pressure to do whatever may be the right thing. That is not necessarily true at all. Having no metrics at all, just like having no government, means that you must fill the void with something. You must decide for yourself how to act, probably based on a subjective judgement of what the situation is combined with your personal values.
      - perl4ever 7y ago
        "Having no metrics at all, just like having no government, means that you must fill the void with something." How can one "fill the void" without measuring anything? Personal values may vary, but I can't see the concept of determining the consequences of your actions as being an optional one of them. It seems to me that if you are definitely rejecting that, then you have no personal values. Even if you say you do.

mlthoughts2018 7y ago

I understand what you mean, but the problem is that first person accounts are so idiosyncratic and situation-specific. It takes a lot of discipline to distill only the most well-supported high level themes from a set of anecdotal qualitative data. The discussion also is at high risk of becoming politicized, where more credence will be given to salient first-person accounts that happen to fit well into some prevalent social narrative of the times. If you can prove that a lot of care was taken to avoid these types of bias in extracting summaries from a collection of qualitative data, then it can be valuable. But often this means you have to be so cautious and err on the side of rejecting all but the most consistent summaries from the data that you run afoul of critics who want the data to be used to support their preferred political narrative. This risk of subverting qualitative data like this into a disingenuous claim that something is data-driven is really high, in addition to all the usual cognitive biases that lead people to read what they want into narrative accounts.

inimino 7y ago
> This risk of subverting qualitative data like this into a disingenuous claim that something is data-driven is really high, in addition to all the usual cognitive biases that lead people to read what they want into narrative accounts. We have engaged on related ideas before[1], and I think your position represents something that I find quite common and deeply unsettling about modern tech culture, but which I don't yet feel I can fully characterize. There is something about the need for everything to be "data-driven" and metrics-focused that I think represents an abrogation of moral responsibility to form, apply, and maintain our own value systems. It is easier and we would apparently rather assign this messy work to some external process or algorithm or metric, which can then be argued to be perfectly rational, correct, data-driven, unbiased[2]. To me this is a dangerous fantasy, closely related to scientism or the general failure to accept any reality outside of what can be measured, counted, or quantified. This is a characteristic failure (in my opinion) of people in their twenties and of STEM majors in our times, so it's not surprising that it would be a characteristic failure of tech, where both those categories are over-represented. It also seems correlated with a general ignorance of and arrogance towards the humanities among tech workers. [1]: https://news.ycombinator.com/item?id=21097061 https://news.ycombinator.com/item?id=21097061 [2]: A related symptom of this disease is in hiring, where companies come up with systems that look very logical and bias free on paper, for example having "objective" scoring criteria and technical challenges (which end up being almost random number generators, but with some bias that tends to reinforce existing imbalances in the industry, e.g. towards recent graduates), and equally "objective" ways of combining results from hiring committee members or interviewers that tend to create and enforce conformity and exclude divergent viewpoints, while giving the appearance of being bias-free, data-driven, and rational. In fact it's a great way that entire companies fool themselves and come up with terrible processes, and then they whine about how hard it is to maintain diversity (of gender, age, whatever) in their ranks. The people that design these systems aren't doing this as some kind of great conspiracy; as far as I can see, they're not evil, just blind.
- perl4ever 7y ago
  "an abrogation of moral responsibility to form, apply, and maintain our own value systems." I think it's a big (and almost certainly invalid) assumption that in the absence of metrics, people "apply and maintain" a consistent value system. It seems to me inconsistent with a belief in the destructive effect of the wrong metrics. If you think that metrics lead to bad results, it's completely inconsistent to suggest that not having metrics is better. When you have metrics, the harm comes from people ignoring what really should be optimized in favor of what is being measured. If you don't have any metrics, then you're only decreasing the pressure to do whatever may be the right thing. It's like, no matter how bad the government is, the only solution is better government, not eliminating it. So, I think your general attitude is ironically capturing the essence of a common intellectual failing of the people you are stereotyping. You know, the glib libertarian tech worker.
  - inimino 7y ago
    > in the absence of metrics, people "apply and maintain" a consistent value system What on earth makes you think I'm making this assumption? I never said anything of the sort. I said that people are abrogating their responsibility to do difficult work (figuring out if the ship is going in the right direction or not) by trying to offload this onto metrics, which are theoretically objective and unbiased. Nowhere did I ever say anything about what people not over-using metrics are doing. > If you don't have any metrics, then you're only decreasing the pressure to do whatever may be the right thing. That is not necessarily true at all. Having no metrics at all, just like having no government, means that you must fill the void with something. You must decide for yourself how to act, probably based on a subjective judgement of what the situation is combined with your personal values.
    - perl4ever 7y ago
      "Having no metrics at all, just like having no government, means that you must fill the void with something." How can one "fill the void" without measuring anything? Personal values may vary, but I can't see the concept of determining the consequences of your actions as being an optional one of them. It seems to me that if you are definitely rejecting that, then you have no personal values. Even if you say you do.