4 ms·

The council-level Eagle Board of Reviewers likely didn't know the individual scouts and were simply applying their individual knowledge about the content review

The council-level Eagle Board of Reviewers likely didn't know the individual scouts and were simply applying their individual knowledge about the content reviewed compared to the whole and made, in this case, poor assumptions about the quality of writing they should expect. As it relates to academia today, this is now happening at a larger scale with folks lacking more trust, not having to think critically about the individual, and applying a similar outcome based on this. This all goes back to critical thought, understanding your audience, and using tools/knowledge in a way it shouldn't be used; it's just now being used more often and with less control than ever. To address your specific comment that those folks may not have been qualified to assess scouts attempting to obtain Eagle, I cannot speak to that, but at the face of it, it would absolutely seem they were definitely not exposed to enough quality writing nor did they have the wherewithal to make decisions at the level they were tasked. Working in the business world, I am continually faced with a wide swath of college-educated adults who simply are unable to write coherently, so none of it surprises me today, but it certainly did when I was 15-16 and thought adults were more capable than actually were/are. I still hate the world.

godelski 3y ago
I've started referring to this phenomena as "Goodhart's Hell." We're living in a world with quickly increasing complexity. Variance in samples is much larger because our reach in what we sample is larger (we talk to people far from our homes, with different cultures, interpretations of words, language patterns, behaviors, ethics, etc). But we've responded to this only by simplifying our evaluations. We've created more tools and metrics that can help guide us, but we don't use them as guides but answers. We're using mathematics/technology not as a tool or language, but a black box. A black box with high accuracy too! Noise/uncertainty has increased but so has our confidence in our answers. It is entirely unwarranted. In essence, we are just hacking metrics and over-fitting our models to increase laziness and comfort. Our ever pursuit of meritocracy ironically takes us further from achieving the goals because we are unwilling to embrace the noise that is inherent to the system, believing that the noise is a lie, believing we can accurately measure everything to absolute (or even meaningful) precision. We have not only abandoned nuance but have actively turned against it and I do not know why. I think it deserves the hell in its name.
- fluoridation 3y ago
  In part it's due to requirements of transparency. If some stranger is going to be evaluating you, a simple way to ensure that you're not being evaluated unfairly is to have the evaluation criteria be very simple, almost to the point of being mechanistic. If you're part of a small enough community it's likely that your evaluator, their overseer, and your peers all know you, so it's very difficult for unfairness to creep in unnoticed. If your community is gigantic everyone starts to be strangers to everyone else, so people can't intuitively know whether someone is being treated unfairly or not. When everyone knows each other it's much more difficult to game the system, because people can tell when you've been lazying around.
  - godelski 3y ago
    I don't disagree with this response. It's actually a good one. But I think we need to be really aware of the trade-offs here. My problem is that people use these metrics as if they are hard proof of competence. But the truth is you have to pick your noise: nepotism or lazy evaluation. Either way, you aren't getting rid of noise. I actually think scale is part of the problem here. In small communities there is more natural nuance and correction as there's high accountability. But with scale accountability often disappear as you disappear into the crowd. And you'll always be able to point at winners and losers to whichever noise you favor. The problem is we're not looking at things holistically and ensuring that our metrics are aligned with our desired outcomes. Instead we often pick a metric because it reasonably looks like it should align with the desired outcome (not proven, just intuited) and then just tell anyone that disagrees that it is obvious instead of acknowledging the downsides. Every metric has limitations. Ignoring them is the problem.
lozenge 3y ago
But, they don't have any knowledge. They don't know how many kids got parents' help and weren't caught, or how many of the kids they "caught" really did know how to write at that level.

godelski 3y ago

I've started referring to this phenomena as "Goodhart's Hell." We're living in a world with quickly increasing complexity. Variance in samples is much larger because our reach in what we sample is larger (we talk to people far from our homes, with different cultures, interpretations of words, language patterns, behaviors, ethics, etc). But we've responded to this only by simplifying our evaluations. We've created more tools and metrics that can help guide us, but we don't use them as guides but answers. We're using mathematics/technology not as a tool or language, but a black box. A black box with high accuracy too! Noise/uncertainty has increased but so has our confidence in our answers. It is entirely unwarranted. In essence, we are just hacking metrics and over-fitting our models to increase laziness and comfort. Our ever pursuit of meritocracy ironically takes us further from achieving the goals because we are unwilling to embrace the noise that is inherent to the system, believing that the noise is a lie, believing we can accurately measure everything to absolute (or even meaningful) precision. We have not only abandoned nuance but have actively turned against it and I do not know why. I think it deserves the hell in its name.

fluoridation 3y ago
In part it's due to requirements of transparency. If some stranger is going to be evaluating you, a simple way to ensure that you're not being evaluated unfairly is to have the evaluation criteria be very simple, almost to the point of being mechanistic. If you're part of a small enough community it's likely that your evaluator, their overseer, and your peers all know you, so it's very difficult for unfairness to creep in unnoticed. If your community is gigantic everyone starts to be strangers to everyone else, so people can't intuitively know whether someone is being treated unfairly or not. When everyone knows each other it's much more difficult to game the system, because people can tell when you've been lazying around.
- godelski 3y ago
  I don't disagree with this response. It's actually a good one. But I think we need to be really aware of the trade-offs here. My problem is that people use these metrics as if they are hard proof of competence. But the truth is you have to pick your noise: nepotism or lazy evaluation. Either way, you aren't getting rid of noise. I actually think scale is part of the problem here. In small communities there is more natural nuance and correction as there's high accountability. But with scale accountability often disappear as you disappear into the crowd. And you'll always be able to point at winners and losers to whichever noise you favor. The problem is we're not looking at things holistically and ensuring that our metrics are aligned with our desired outcomes. Instead we often pick a metric because it reasonably looks like it should align with the desired outcome (not proven, just intuited) and then just tell anyone that disagrees that it is obvious instead of acknowledging the downsides. Every metric has limitations. Ignoring them is the problem.

lozenge 3y ago

But, they don't have any knowledge. They don't know how many kids got parents' help and weren't caught, or how many of the kids they "caught" really did know how to write at that level.