4 ms·

This is the age-old doctor question. You go to the doctor's office because you have a medical problem. Doctor 1 comes in on-time, she's smart, capable, friendl

This is the age-old doctor question. You go to the doctor's office because you have a medical problem. Doctor 1 comes in on-time, she's smart, capable, friendly, and spends a good amount of time concerned about you and your situation. Doctor 2 comes in late, she's irascible, pedantic, hurried, and only listens long enough to tell you what to do. She also insults you and your family in various ways and doesn't seem to care. After the visit, you go to Yelp. Which one was the better doctor? You have no idea! But do you think that's going to stop somebody from forming and posting an opinion anyway? You can make the argument that these other qualities are good to have in a doctor. I agree. I want a doctor like the first one. But people go to doctors first and foremost to get better, not to make new social acquaintances. If the first doc is incompetent and hurting people, and the second doc a genius, and her attitude is actually causing more people to comply with what's best for them out of a sense anger at her style, perhaps? That's the thing. Who the hell cares if the outside looks cool/sexy/friendly or not? So when we ask for reviews from students who have never worked in the field they're studying for, they're like that Yelp doctor reviewer. I don't see how this situation is good for anybody. In fact, it'll probably lead to a bunch friendly, good-looking morons with great people skills entertaining bunch of kids who aren't learning anything. That's the only natural consequence here.

pas 8y ago
It's not like there can't be an accurate way to measure this. Pre-register patient problems, and see how fast doctors solve problems on average/median. Then it doesn't matter how jovial/amicable the doc is. With enough data you can fish out potential signals (and then test them properly), maybe bedside manner really doesn't matter, maybe being on time is more important. (Because then people spend less time in the clinic among other sick people.) And as others mentioned the teacher eval problem can be solved by splitting the lecturer, exam writer and grader roles. (Grading should be done by a book written as edge cases accumulate, edge cases should be handled without names to prevent bias, as much as possible.) And then aggregate stats can be published about each class and school.
- DanielBMarkham 8y ago
  Sorry no, it doesn't work like that. In either case. Doctors, like professors, are evaluated by former patients based on a mix of factors. In your doctor example, doctors don't "solve problems". They see patients. Many times these patients present with conflicting and hard-to-resolve symptoms. I imagine with the internet this has gotten worse. They use a mix of Occam's Razor, deductive reasoning, and social psychology for the patient's long-term health. You don't want a doctor that finds the answers to complex problems and gives out morphine. You want a doctor that over decades has patients with the best outcomes. In my example, the one doctor may have ticked you off so much you decide never to go back -- and quit smoking just to prove her wrong. That's a win for both you and the doc, although there's no problem being solved. You can speculate various ways to go at this. Perhaps the doctor is evaluated by other doctors working at that location. Or perhaps it's a matrix of various things. As long as you're just pulling stuff out of the air, you can invent any kind of system you'd like. And tech will deliver it for you. Human interventions are messy affairs, and thank goodness for that. We are not robots. Your proposal might do a great job of sorting out professors who teach students well enough to take the test, but heck if I see where that is the end of the analysis. That's just the beginning. We're just getting started talking about professors who can lead a class to mostly pass a test. That should be a baseline for any professor. This is the problem we see in software development. You can come up with all sorts of metrics to stand in place of evaluating whether a team/person is good or not, but whatever you come up with mostly seem to work -- and has very bad effects on those being measured. (With the possible exception of business-value-loop feedback time) I don't think this is tractable in the way you seem to want it. If I were picking a college or professor, I might look at five or six variables and weigh them in various ways depending on my personal needs and goals at the time -- which might change tomorrow. I might want to go interview them. Meet some current and former students. There is a severe and onerous "platform danger" in tech. It starts with the conceit that we can build platforms for anything. It ends up with people not-as-smart-as-they-used-to-be having Google tell them where to go for dinner. Or Wally in Dilbert coding himself up a minivan. [0] I'm not saying the metric definitions and accumulations aren't interesting. They deserve to be considered. I'm saying that by reducing human judgment to single vector, it makes us as a species dumber. That's one of those intuitive things that sound good but are in reality quite bad. 0. https://dilbert.com/strip/1995-11-13 https://dilbert.com/strip/1995-11-13
sethammons 8y ago
Our evals led to the dismissal of a professor. She was absolutely gorgeous, and a delight to talk to. And couldn't convey account proceedures worth a damn. That is what the students cared about in our case.

pas 8y ago

It's not like there can't be an accurate way to measure this. Pre-register patient problems, and see how fast doctors solve problems on average/median. Then it doesn't matter how jovial/amicable the doc is. With enough data you can fish out potential signals (and then test them properly), maybe bedside manner really doesn't matter, maybe being on time is more important. (Because then people spend less time in the clinic among other sick people.) And as others mentioned the teacher eval problem can be solved by splitting the lecturer, exam writer and grader roles. (Grading should be done by a book written as edge cases accumulate, edge cases should be handled without names to prevent bias, as much as possible.) And then aggregate stats can be published about each class and school.

DanielBMarkham 8y ago
Sorry no, it doesn't work like that. In either case. Doctors, like professors, are evaluated by former patients based on a mix of factors. In your doctor example, doctors don't "solve problems". They see patients. Many times these patients present with conflicting and hard-to-resolve symptoms. I imagine with the internet this has gotten worse. They use a mix of Occam's Razor, deductive reasoning, and social psychology for the patient's long-term health. You don't want a doctor that finds the answers to complex problems and gives out morphine. You want a doctor that over decades has patients with the best outcomes. In my example, the one doctor may have ticked you off so much you decide never to go back -- and quit smoking just to prove her wrong. That's a win for both you and the doc, although there's no problem being solved. You can speculate various ways to go at this. Perhaps the doctor is evaluated by other doctors working at that location. Or perhaps it's a matrix of various things. As long as you're just pulling stuff out of the air, you can invent any kind of system you'd like. And tech will deliver it for you. Human interventions are messy affairs, and thank goodness for that. We are not robots. Your proposal might do a great job of sorting out professors who teach students well enough to take the test, but heck if I see where that is the end of the analysis. That's just the beginning. We're just getting started talking about professors who can lead a class to mostly pass a test. That should be a baseline for any professor. This is the problem we see in software development. You can come up with all sorts of metrics to stand in place of evaluating whether a team/person is good or not, but whatever you come up with mostly seem to work -- and has very bad effects on those being measured. (With the possible exception of business-value-loop feedback time) I don't think this is tractable in the way you seem to want it. If I were picking a college or professor, I might look at five or six variables and weigh them in various ways depending on my personal needs and goals at the time -- which might change tomorrow. I might want to go interview them. Meet some current and former students. There is a severe and onerous "platform danger" in tech. It starts with the conceit that we can build platforms for anything. It ends up with people not-as-smart-as-they-used-to-be having Google tell them where to go for dinner. Or Wally in Dilbert coding himself up a minivan. [0] I'm not saying the metric definitions and accumulations aren't interesting. They deserve to be considered. I'm saying that by reducing human judgment to single vector, it makes us as a species dumber. That's one of those intuitive things that sound good but are in reality quite bad. 0. https://dilbert.com/strip/1995-11-13 https://dilbert.com/strip/1995-11-13

sethammons 8y ago

Our evals led to the dismissal of a professor. She was absolutely gorgeous, and a delight to talk to. And couldn't convey account proceedures worth a damn. That is what the students cared about in our case.