6 ms·

Bias in student evaluations of teaching

30 points by mariedm 8y ago

mcherm 8y ago
Why do I keep encountering people who apparently think that Twitter is an appropriate medium for conveying thoughtful, considered information?
- rhacker 8y ago
  Because they are trying to get the highest number of people to see it - and that is Twitter, not a link on twitter, but Twitter.
- xena 8y ago
  Because people don't click links
- olliej 8y ago
  I have wondered about that recently as well - I assume it's because "everyone" already has a twitter account, so if you do have some one off occasion that you want to say a pile of stuff, you already have that account that is easily sharable. Any other blog like service requires you signing up for another account, and sharing your email address again. So you use twitter. The next time you want to say something, you still don't have a blog account so you take the same logic and end up posting through twitter again. Things like threader seem to make it easier for your thread to be shared as a single document now, which makes it less annoying to share and/or read. Then, with the death of Reader no one really has a non-FB, non-Twitter style mechanism for following people any more so if you want to say something and encourage people to follow you then twitter is your best bet. Likewise if you want to hear what other people say (ugh, internet commentators ;) ) twitter makes it possible to easily get replies and you don't need to worry about comment spam continuously appearing.
- tptacek 8y ago
  Because, for all its flaws, it is quite effective at conveying thoughtful, considered information.
- yholio 8y ago
  It's the secret sauce of all social networks, users are strongly encouraged to create and distribute content on the platform itself, as opposed to talking about external content. They use dark UI flows, subtle psychological tricks and emotional rewards, everything to abstract the real world away and presume the only audience that matters is that within the walls.
adamnemecek 8y ago
Are students of color harsher on instructors of color as well?
- olliej 8y ago
  Unsure, but given this thread is mostly reporting on bias against women instructors it's not a super relevant question.
  - deleted 8y ago
    [deleted]
- rhacker 8y ago
  Most evaluations likely don't ask for that, so they probably don't have that information right now.
- roenxi 8y ago
  Just to throw in some context, unless I've read the report wrong [0], there is evidence that the bias against female TAs was led by female students, and the bias by male students was difficult to detect with statistical significance. [0] https://www.scienceopen.com/document_file/25ff22be-8a1b-4c97-9d88-084c8d98187a/ScienceOpen/3507_XE6680747344554310733.pdf https://www.scienceopen.com/document_file/25ff22be-8a1b-4c97... Tables 8, 9 and 10.
ryanwaggoner 8y ago
I know how this is going to sound, but I'm genuinely curious: how do they measure whether the evaluations are gender biased, or whether the quality of the instruction is not equally distributed between the genders? Before I get a bunch of hate and downvotes, I'm not suggesting that female teachers get lower evaluation scores because they're not as good on average as their male counterparts. But I assume that researchers can show why that's not the underlying source of the difference in ratings, since that would be an obvious objection to these findings. Just curious how they separate that out, that's all. EDIT: nevermind, this was just laziness on my part. Here's the study, and it's actually not that hard to understand: https://www.scienceopen.com/document_file/25ff22be-8a1b-4c97-9d88-084c8d98187a/ScienceOpen/3507_XE6680747344554310733.pdf https://www.scienceopen.com/document_file/25ff22be-8a1b-4c97... TLDR: In one experiment, they randomly assigned students to a TA and compared how well the students did in the class to determine whether the instructors were of comparable quality. In another experiment they secretly switched the TA identities in an online course so the students who had the female TA thought they had the male TA, and vice versa. Very nice.
- trjordan 8y ago
  Did you read the linked papers? They go into a lot of detail about how they did it.
  - ryanwaggoner 8y ago
    Did you read my comment? I know yours wasn’t there when I added that edit :)
ballenf 8y ago
Would it be that hard to judge professors based on students success (in subsequent classes) using something like a pagerank algorithm? But that wouldn't necessarily correlate with institution popularity and success. Academia seems to be becoming more of a customer service-oriented business where customer (student) perception is more important than the value/quality of the service provided.
- deleted 8y ago
  [deleted]
- eesmith 8y ago
  It's called "Value Added Method" (VAM) and is used in many school systems. It's also unreliable, says the American Statistical Association https://www.educationdive.com/news/statisticians-reject-value-added-teacher-evaluations/250785/ https://www.educationdive.com/news/statisticians-reject-valu... or original ASA statement at https://www.amstat.org/asa/files/pdfs/POL-ASAVAM-Statement.pdf https://www.amstat.org/asa/files/pdfs/POL-ASAVAM-Statement.p... . Details about the difficulties in the implementation, at http://vamboozled.com/ http://vamboozled.com/ , which as you can infer is against VAM. Eg: * Class composition varied so much that comparisons of value-added scores of two teachers were only valid if both teachers are assigned students with similar characteristics. * Annual fluctuations in results were so large that they lead to widely varying conclusions from one year to the next for the same teacher. * There was strong evidence that results were often due to the teaching environment, not just the teacher.
olliej 8y ago
A really simple indicator of this the long term relevance of "hotness" in things like ratemyprofessor - mercifully they killed that, but it should never have been present in the first place - which showed that "hotness" was strongly correlated to a professor or TAs rating. Couple with other studies that hotness ratings of women are more extreme (tending towards a binomial hot vs. not), and more negative (I would assume, but am not sure of actual real study to back it up, because "average" hotness gets demoted).
- philipodonnell 8y ago
  I'm kind of conflicted about that. A strongly explanatory variable is valuable to have in order to control for its effects. Voluntarily removing it reduces the usefulness of the derived statistics and in this case I guess seems to make ratemyprofessor just a niche version of hotornot. Perhaps a middle ground would have been to allow students to rate "hottness" and explicitly controlled for it when ranking professors, but never making a given professor's "average hotness" viewable to users?
golemiprague 8y ago
So it is not only gender and colour, it is also age, beauty and million of other parameters that usually affect how people relate to other people. That's how life are, it is not going to change, you can't be conscious about all those parameters all the time and adjust evaluations, how many points are you going to add per one year of age or one shade of darker colour ?
bluGill 8y ago
I know I tended to rate my foreign" instructors poorly. Not that I want to be dislike them, but when I have trouble understanding their accent it is hard to learn from them. If I ever have to give a lecture in a foreign language I'm sure my students to have the same reaction (of course as a human you can be sure that if I actually spoke a language well enough for this to be reasonable I'd consider my accent perfect)
- deleted 8y ago
  [deleted]
btilly 8y ago
I knew one CS professor who always had very good evaluations. His trick? He brought free cookies on the day he was going to get evaluated. He swore that it worked.
shittyadmin 8y ago
Is there an actual solution proposed here or is this just full of hot air? He says adjusting can't work and that explaining bias doesn't do the job, so what is he suggesting? Don't consider a professors performance at all? Add some sort of standardized testing system?
AlexCoventry 8y ago
Worth noting that all four links in his thread are to the same paper.
slededit 8y ago
This happens in reverse too. Teachers assume high achieving students will continue to achieve and give them more slack. The trick to all my "writing" classes was to simply put the most effort in the first assignment. It was pretty clear that future marks were a function of prior marks, and I tested this hypothesis in different classes by modulating my effort on various assignments throughout the year.
analog31 8y ago
I spent one semester teaching at a major state university, many years ago. Here are some anecdata: * I shared an office with a guy who was team-teaching a course with a professor. My office mate was an adjunct, the professor was coming up for tenure. They fought bitterly over the grading of the course. The prof wanted to give easy grades because he needed to boost his student evaluation scores. * It was common knowledge that the best predictor of evaluation scores was the grades that the students expected to receive. It was as if the system was rigged to encourage grade inflation. That was 20 years ago. * Female teachers received in their evaluations, among other things, violent threats.
lacker 8y ago
Education is screwed up in many ways. Colleges would rather look good in the US News rankings than educate their students well. Students would rather have attractive professors than ones who teach them effectively. And yet the whole thing is massively popular and expensive.

30 points by mariedm 8y ago

Why do I keep encountering people who apparently think that Twitter is an appropriate medium for conveying thoughtful, considered information?

rhacker 8y ago
Because they are trying to get the highest number of people to see it - and that is Twitter, not a link on twitter, but Twitter.
xena 8y ago
Because people don't click links
olliej 8y ago
I have wondered about that recently as well - I assume it's because "everyone" already has a twitter account, so if you do have some one off occasion that you want to say a pile of stuff, you already have that account that is easily sharable. Any other blog like service requires you signing up for another account, and sharing your email address again. So you use twitter. The next time you want to say something, you still don't have a blog account so you take the same logic and end up posting through twitter again. Things like threader seem to make it easier for your thread to be shared as a single document now, which makes it less annoying to share and/or read. Then, with the death of Reader no one really has a non-FB, non-Twitter style mechanism for following people any more so if you want to say something and encourage people to follow you then twitter is your best bet. Likewise if you want to hear what other people say (ugh, internet commentators ;) ) twitter makes it possible to easily get replies and you don't need to worry about comment spam continuously appearing.
tptacek 8y ago
Because, for all its flaws, it is quite effective at conveying thoughtful, considered information.
yholio 8y ago
It's the secret sauce of all social networks, users are strongly encouraged to create and distribute content on the platform itself, as opposed to talking about external content. They use dark UI flows, subtle psychological tricks and emotional rewards, everything to abstract the real world away and presume the only audience that matters is that within the walls.

adamnemecek 8y ago

Are students of color harsher on instructors of color as well?

olliej 8y ago
Unsure, but given this thread is mostly reporting on bias against women instructors it's not a super relevant question.
- deleted 8y ago
  [deleted]
rhacker 8y ago
Most evaluations likely don't ask for that, so they probably don't have that information right now.
roenxi 8y ago
Just to throw in some context, unless I've read the report wrong [0], there is evidence that the bias against female TAs was led by female students, and the bias by male students was difficult to detect with statistical significance. [0] https://www.scienceopen.com/document_file/25ff22be-8a1b-4c97-9d88-084c8d98187a/ScienceOpen/3507_XE6680747344554310733.pdf https://www.scienceopen.com/document_file/25ff22be-8a1b-4c97... Tables 8, 9 and 10.

ryanwaggoner 8y ago

I know how this is going to sound, but I'm genuinely curious: how do they measure whether the evaluations are gender biased, or whether the quality of the instruction is not equally distributed between the genders? Before I get a bunch of hate and downvotes, I'm not suggesting that female teachers get lower evaluation scores because they're not as good on average as their male counterparts. But I assume that researchers can show why that's not the underlying source of the difference in ratings, since that would be an obvious objection to these findings. Just curious how they separate that out, that's all. EDIT: nevermind, this was just laziness on my part. Here's the study, and it's actually not that hard to understand: https://www.scienceopen.com/document_file/25ff22be-8a1b-4c97-9d88-084c8d98187a/ScienceOpen/3507_XE6680747344554310733.pdf https://www.scienceopen.com/document_file/25ff22be-8a1b-4c97... TLDR: In one experiment, they randomly assigned students to a TA and compared how well the students did in the class to determine whether the instructors were of comparable quality. In another experiment they secretly switched the TA identities in an online course so the students who had the female TA thought they had the male TA, and vice versa. Very nice.

trjordan 8y ago
Did you read the linked papers? They go into a lot of detail about how they did it.
- ryanwaggoner 8y ago
  Did you read my comment? I know yours wasn’t there when I added that edit :)

ballenf 8y ago

Would it be that hard to judge professors based on students success (in subsequent classes) using something like a pagerank algorithm? But that wouldn't necessarily correlate with institution popularity and success. Academia seems to be becoming more of a customer service-oriented business where customer (student) perception is more important than the value/quality of the service provided.

deleted 8y ago
[deleted]
eesmith 8y ago
It's called "Value Added Method" (VAM) and is used in many school systems. It's also unreliable, says the American Statistical Association https://www.educationdive.com/news/statisticians-reject-value-added-teacher-evaluations/250785/ https://www.educationdive.com/news/statisticians-reject-valu... or original ASA statement at https://www.amstat.org/asa/files/pdfs/POL-ASAVAM-Statement.pdf https://www.amstat.org/asa/files/pdfs/POL-ASAVAM-Statement.p... . Details about the difficulties in the implementation, at http://vamboozled.com/ http://vamboozled.com/ , which as you can infer is against VAM. Eg: * Class composition varied so much that comparisons of value-added scores of two teachers were only valid if both teachers are assigned students with similar characteristics. * Annual fluctuations in results were so large that they lead to widely varying conclusions from one year to the next for the same teacher. * There was strong evidence that results were often due to the teaching environment, not just the teacher.

olliej 8y ago

A really simple indicator of this the long term relevance of "hotness" in things like ratemyprofessor - mercifully they killed that, but it should never have been present in the first place - which showed that "hotness" was strongly correlated to a professor or TAs rating. Couple with other studies that hotness ratings of women are more extreme (tending towards a binomial hot vs. not), and more negative (I would assume, but am not sure of actual real study to back it up, because "average" hotness gets demoted).

philipodonnell 8y ago
I'm kind of conflicted about that. A strongly explanatory variable is valuable to have in order to control for its effects. Voluntarily removing it reduces the usefulness of the derived statistics and in this case I guess seems to make ratemyprofessor just a niche version of hotornot. Perhaps a middle ground would have been to allow students to rate "hottness" and explicitly controlled for it when ranking professors, but never making a given professor's "average hotness" viewable to users?

golemiprague 8y ago

So it is not only gender and colour, it is also age, beauty and million of other parameters that usually affect how people relate to other people. That's how life are, it is not going to change, you can't be conscious about all those parameters all the time and adjust evaluations, how many points are you going to add per one year of age or one shade of darker colour ?

bluGill 8y ago

I know I tended to rate my foreign" instructors poorly. Not that I want to be dislike them, but when I have trouble understanding their accent it is hard to learn from them. If I ever have to give a lecture in a foreign language I'm sure my students to have the same reaction (of course as a human you can be sure that if I actually spoke a language well enough for this to be reasonable I'd consider my accent perfect)

deleted 8y ago
[deleted]

btilly 8y ago

I knew one CS professor who always had very good evaluations. His trick? He brought free cookies on the day he was going to get evaluated. He swore that it worked.

shittyadmin 8y ago

Is there an actual solution proposed here or is this just full of hot air? He says adjusting can't work and that explaining bias doesn't do the job, so what is he suggesting? Don't consider a professors performance at all? Add some sort of standardized testing system?

AlexCoventry 8y ago

Worth noting that all four links in his thread are to the same paper.

slededit 8y ago

This happens in reverse too. Teachers assume high achieving students will continue to achieve and give them more slack. The trick to all my "writing" classes was to simply put the most effort in the first assignment. It was pretty clear that future marks were a function of prior marks, and I tested this hypothesis in different classes by modulating my effort on various assignments throughout the year.

analog31 8y ago

I spent one semester teaching at a major state university, many years ago. Here are some anecdata: * I shared an office with a guy who was team-teaching a course with a professor. My office mate was an adjunct, the professor was coming up for tenure. They fought bitterly over the grading of the course. The prof wanted to give easy grades because he needed to boost his student evaluation scores. * It was common knowledge that the best predictor of evaluation scores was the grades that the students expected to receive. It was as if the system was rigged to encourage grade inflation. That was 20 years ago. * Female teachers received in their evaluations, among other things, violent threats.

lacker 8y ago

Education is screwed up in many ways. Colleges would rather look good in the US News rankings than educate their students well. Students would rather have attractive professors than ones who teach them effectively. And yet the whole thing is massively popular and expensive.