3 ms·

All I want, is fair and transparent system that allows me to compete with people from $top_school. Why somebody from $top_school should have an advantage just

• godelski 3y ago
  Of course want a fair system, but I'm trying to explain why a fair system is quite difficult to create. It isn't about perfection getting in the way of good enough, it is that we've already been trying your suggestion and that's how we've ended up with the system we're at. To be clear, there is a lot of bias in standardized tests. There's also, as we've discussed, not a strong correlation with test performance and work performance. But I don't just preach this, I practice this. When I teach I make my students do projects. The projects are far more important than the single test I have (to fit department suggestions). I give them lots of freedom and let them explore. Basically I treat them like junior engineers but let them pick their projects. Honestly, I see all aspects of students improve through this rather than when I tried testing or homeworks because they are more likely to find motivation. Because memory is based on how useful you think something is. Granted I teach a popular class but still. I can definitely judge a student's performance and how much they've learned FAR better from a project or two than from any amount of testing or homeworks. They just don't even compare in ability to evaluate. Problem is, there's absolutely no way to standardize such a thing. In fact, I'm literally arguing that the problem here is that these aspects are diametrically opposed. The thing is that what sounds nice and "should work" often isn't. It's also often the thing people have already tried. You have to actually embrace the noise and complexity of the environment otherwise you just get hacking. I know it sucks, but you sometimes gotta do things the hard way.
  • tester756 3y ago
    >To be clear, there is a lot of bias in standardized tests Is there? I'd say that there is very, very minimal amount of bias in them. >When I teach I make my students do projects. The projects are far more important than the single test I have Projects are better, but you cannot do them at scale and compare them.
    • godelski 3y ago
      > Is there? I'd say that there is very, very minimal amount of bias in them. Yes, it is the exact thing we've been talking about since standardized testing is the status quo. There are also plenty of studies on this if you'd like to google for more. It's also important to remember that there are very clear test taking skills, and that these standardized tests intentionally try to trick students (you'll even see mentions of this in your test prep books). > Projects are better, but you cannot do them at scale and compare them. Unfortunately some things don't scale very well. Best you can do is scale by hierarchy. Using local groups. But before you say that this requires more trust I'd also note that we do require a lot of trust in our society and for the most part people are pretty trustworthy. It is when metric hacking is encouraged that trust is lost. A lot of standardized testing has a lot of trust built in as well. But there's another way to think about it. If at the local level you are evaluating someone a certain way then those are the skills that you are teaching them. When you are doing standardized testing you are mainly teaching students how to be good test takers. But we've seen that that tends to overfit as any standardized test is incapable of incorporating nuanced thinking due to requiring to allow for high variance in test takers. On the other hand, local evaluations via projects are directly teaching students to learn to do something much closer to what they'll do in the real world. Also tends to have larger benefits in keeping students excited/motivated as well as find their direction. But yes, this requires more hierarchical trust. But this kinda already exists. We already rank schools, we just tend to do this by standardized test scores. But you can use other metrics as well. The schools could even improve by weighting teachers differently. But my main point here is that the main benefit here isn't actually the evaluation ability at the abstracted level (the level bureaucrats want, the level we judge populations, the aggregate levels -- which I am actually arguing are extremely noisy predictors (read: unreliable). Aggregate metrics are often unreliable fwiw: see {Simpson's,Berkson's} Paradox). Means often get you into trouble), but rather that if in local cliques you enforce good metrics that this propagates upwards. Think like Conway's game (or even physics): if your local set of rules is good, you can generate arbitrary complexity at the large scale due to compounding effects. To be clear: the goals I'd want schooling to align with are: 1) maximizing education of people (both specific to their desired specialization as well as general knowledge and skills to learn to learn), 2) achieve threshold value for socialization level (ensuring people can interact in society. This is often implicit but I'm making explicit), 3) maximizing labor efficiency when student enters the workforce (minimize onboard training). This is an incomplete list and should be thought out more and far more carefully so we can describe the nonlinear program, but this is a good enough outline -- at least for HN. But I'll also add that there's a certain level of noise that is quite hard to get rid of. To truly evaluate good teaching methods with good accuracy you actually need to track students over very long periods of time. Which gets extremely difficult to model since those students take varying paths and interact with a lot of different teachers who you must all decouple. You're now essentially trying to solve a problem similar to neural network interpretation/explainability. This is true for no matter what evaluation method you use. TLDR: metrics are fucking hard, do not underestimate the difficulty of evaluation. It'll bite you in the ass