8 ms·

Since it's a multiple choice question, one assumes there is a correct answer, and presumably that is A, because there is a clear argument for it (you wish to sa

Since it's a multiple choice question, one assumes there is a correct answer, and presumably that is A, because there is a clear argument for it (you wish to sample from the population distribution, which can be viewed as a mixture of an initial-respondents distribution and an initial-non-respondents distribution and so you cannot claim to have sampled from the mixture until you've sampled from both mixture components, since they are probably statistically distinct). Tangentially though, a big question here is how wise it is for the US education system to be based so heavily on multiple choice questions. There are other countries with decent education systems that do not do this.

pdonis 4y ago
> one assumes there is a correct answer, and presumably that is A, because there is a clear argument for it (you wish to sample from the population distribution, which can be viewed as a mixture of an initial-respondents distribution and an initial-non-respondents distribution and so you cannot claim to have sampled from the mixture until you've sampled from both mixture components, since they are probably statistically distinct) But you can't survey non-respondents--because they don't respond! What option A will actually result in is having a mixture of people who responded on the first try, and people who responded on the second try. In principle, neither of those will be representative of non-respondents. Whether this creates a significant problem in practice will depend on lots of other assumptions. My own first choice would be either E or D, depending on (a) whether you have enough resources available to send 30 more emails, and (b) how much statistical power you are sacrificing with a sample of only 90 instead of 120. (If you were doing things properly, the number of emails you send out initially would be larger than the actual number you needed to get enough statistical power, by some factor that would depend on what fraction of people you expected to respond. In which case D would be the obvious option.)
- montroser 4y ago
  The idea with A is that you will get a more representative sample if you follow up specifically with the people who did not initially respond. Yes, some will never respond, but the ones that do answer will increase the quality of the data, moreso than an additional random sample. The people who did not initially respond are more likely to be busy/lazy/unengaged, and without as many responses from that cohort, the data will be skewed.
  - pdonis 4y ago
    > The idea with A is that you will get a more representative sample if you follow up specifically with the people who did not initially respond. Perhaps. But you are also creating a potential confounder, since you now have two categories of responders instead of one.
    - ksenzee 4y ago
      That seems like a pretty small potential confounder. It doesn't seem to bother the US Census, for example, that they get some people's response via mail right away, and others they have to go knock on the door and pester them to respond. Any random sample of people is going to include some who really like answering questionnaires, and some who hate it or are too distracted or busy. Designing a survey where you plan to contact people multiple times seems perfectly normal and reasonable.
      - dragonwriter 4y ago
        > That seems like a pretty small potential confounder. It doesn't seem to bother the US Census, for example, that they get some people's response via mail right away, and others they have to go knock on the door and pester them to respond. Any random sample [...] The decennial census is not a random sample, though the Census Bureau does sruveys that are random samples separately.
        Retric 4y ago
        The Census uses this technique when doing random samples. One of the ways to improve accuracy is to put a lot of effort into contacting a random sample of non respondents which simply isn’t viable at scale.
      - pdonis 4y ago
        The US Census is not doing sociological research on people's opinions. They're collecting factual information. Big difference.
tantalor 4y ago
> how wise it is for the US education system to be based so heavily on multiple choice questions Who said anything about "US education system"? Where in the original article does it say "US"?
- yesenadam 4y ago
  Well, the question uses the word "freshmen", which I believe is US-only.
- gpt5 4y ago
  Interestingly, in Israel they call a test with multiple choice questions "an American test".
- Myrmornis 4y ago
  That's the point! Evaluating knowledge of university-level topics with childish multiple choice questions is so strongly associated with the United States that there's not that much doubt it's a US test.
marzell 4y ago
This smells like a scenario where the teacher isn't looking for the "right" answer, speaking broadly or objectively. Instead, I suspect there is a particular concept or perspective they have been trying to impress on their students recently, and in this case they are expecting the answer that most demonstrates they have paid attention to the recent lessons. I could be wrong of course.
Xcelerate 4y ago
> a big question here is how wise it is for the US education system to be based so heavily on multiple choice questions There was a great blog post I read a while back on redesigning multiple choice tests to allow the student to indicate the "confidence" of their response, with a more confident answer being rewarded/penalized more heavily than a response with low confidence. This allowed for a statistically better sample of how well the student learned the material. I thought for sure the post was written by Scott Aaronson, but I haven't been able to find it despite extensively searching his blog, so maybe it was someone else.
- notRobot 4y ago
  Is this this? https://theeffortfuleducator.com/2020/06/22/confidence-weighted-multiple-choice-questioning/ https://theeffortfuleducator.com/2020/06/22/confidence-weigh... Found with the following Google query: "multiple choice" "school" "confidence" "blog"
- smilliken 4y ago
  I believe you're thinking of this post by Terry Tao: https://terrytao.wordpress.com/2016/06/01/how-to-assign-partial-credit-on-an-exam-of-true-false-questions/ https://terrytao.wordpress.com/2016/06/01/how-to-assign-part...
  - Xcelerate 4y ago
    Yep, that was it! Thanks!
- Dylan16807 4y ago
  If I'm taking that test, why would I ever give a confidence estimate other than 0% and 100%? If I think I have a positive expected return, it's worth the risk to go for max points. If I don't, then I don't want to lose less points, I want to not answer at all, or equivalently give it zero weight.
  - yunwal 4y ago
    If you calculated scores using a mean squared error loss function, it'd be better to hedge your bets if you were uncertain.
abruzzi 4y ago
I remember taking a multiple choice question test for some scholarship 35 years ago. 20 questions with the explicit instructions that for any question all of the choices could be correct or none of them or combination, and you were supposed to check only the answers that were correct. It was a VERY hard test. In addition you could score less than zero since every correct item that you checked would add points but every incorrect item itme you checked would subtract points AND every correct item you didn't check would also subtract points. You didn't get points for not checking incorrect items. I've never seen a test like that for school work.
- baq 4y ago
  I had a few of those in university, they were a favorite of a particular professor for a while. On the very first exam, given that you had to get 50% of points to get a passing grade, IIRC about 20% had a score above 0. Those exams got easier after a couple years. I imagine there were some words to be had with the dean.
Someone 4y ago
I would choose A, too, but if you can do A, the questionnaire isn’t anonymous. I would go for A’: email all of them, reminding them to fill in the questionnaire, if they haven’t already done so.

pdonis 4y ago

> one assumes there is a correct answer, and presumably that is A, because there is a clear argument for it (you wish to sample from the population distribution, which can be viewed as a mixture of an initial-respondents distribution and an initial-non-respondents distribution and so you cannot claim to have sampled from the mixture until you've sampled from both mixture components, since they are probably statistically distinct) But you can't survey non-respondents--because they don't respond! What option A will actually result in is having a mixture of people who responded on the first try, and people who responded on the second try. In principle, neither of those will be representative of non-respondents. Whether this creates a significant problem in practice will depend on lots of other assumptions. My own first choice would be either E or D, depending on (a) whether you have enough resources available to send 30 more emails, and (b) how much statistical power you are sacrificing with a sample of only 90 instead of 120. (If you were doing things properly, the number of emails you send out initially would be larger than the actual number you needed to get enough statistical power, by some factor that would depend on what fraction of people you expected to respond. In which case D would be the obvious option.)

montroser 4y ago
The idea with A is that you will get a more representative sample if you follow up specifically with the people who did not initially respond. Yes, some will never respond, but the ones that do answer will increase the quality of the data, moreso than an additional random sample. The people who did not initially respond are more likely to be busy/lazy/unengaged, and without as many responses from that cohort, the data will be skewed.
- pdonis 4y ago
  > The idea with A is that you will get a more representative sample if you follow up specifically with the people who did not initially respond. Perhaps. But you are also creating a potential confounder, since you now have two categories of responders instead of one.
  - ksenzee 4y ago
    That seems like a pretty small potential confounder. It doesn't seem to bother the US Census, for example, that they get some people's response via mail right away, and others they have to go knock on the door and pester them to respond. Any random sample of people is going to include some who really like answering questionnaires, and some who hate it or are too distracted or busy. Designing a survey where you plan to contact people multiple times seems perfectly normal and reasonable.
    - dragonwriter 4y ago
      > That seems like a pretty small potential confounder. It doesn't seem to bother the US Census, for example, that they get some people's response via mail right away, and others they have to go knock on the door and pester them to respond. Any random sample [...] The decennial census is not a random sample, though the Census Bureau does sruveys that are random samples separately.
      - Retric 4y ago
        The Census uses this technique when doing random samples. One of the ways to improve accuracy is to put a lot of effort into contacting a random sample of non respondents which simply isn’t viable at scale.
    - pdonis 4y ago
      The US Census is not doing sociological research on people's opinions. They're collecting factual information. Big difference.

tantalor 4y ago

> how wise it is for the US education system to be based so heavily on multiple choice questions Who said anything about "US education system"? Where in the original article does it say "US"?

yesenadam 4y ago
Well, the question uses the word "freshmen", which I believe is US-only.
gpt5 4y ago
Interestingly, in Israel they call a test with multiple choice questions "an American test".
Myrmornis 4y ago
That's the point! Evaluating knowledge of university-level topics with childish multiple choice questions is so strongly associated with the United States that there's not that much doubt it's a US test.

marzell 4y ago

This smells like a scenario where the teacher isn't looking for the "right" answer, speaking broadly or objectively. Instead, I suspect there is a particular concept or perspective they have been trying to impress on their students recently, and in this case they are expecting the answer that most demonstrates they have paid attention to the recent lessons. I could be wrong of course.

Xcelerate 4y ago

> a big question here is how wise it is for the US education system to be based so heavily on multiple choice questions There was a great blog post I read a while back on redesigning multiple choice tests to allow the student to indicate the "confidence" of their response, with a more confident answer being rewarded/penalized more heavily than a response with low confidence. This allowed for a statistically better sample of how well the student learned the material. I thought for sure the post was written by Scott Aaronson, but I haven't been able to find it despite extensively searching his blog, so maybe it was someone else.

notRobot 4y ago
Is this this? https://theeffortfuleducator.com/2020/06/22/confidence-weighted-multiple-choice-questioning/ https://theeffortfuleducator.com/2020/06/22/confidence-weigh... Found with the following Google query: "multiple choice" "school" "confidence" "blog"
smilliken 4y ago
I believe you're thinking of this post by Terry Tao: https://terrytao.wordpress.com/2016/06/01/how-to-assign-partial-credit-on-an-exam-of-true-false-questions/ https://terrytao.wordpress.com/2016/06/01/how-to-assign-part...
- Xcelerate 4y ago
  Yep, that was it! Thanks!
Dylan16807 4y ago
If I'm taking that test, why would I ever give a confidence estimate other than 0% and 100%? If I think I have a positive expected return, it's worth the risk to go for max points. If I don't, then I don't want to lose less points, I want to not answer at all, or equivalently give it zero weight.
- yunwal 4y ago
  If you calculated scores using a mean squared error loss function, it'd be better to hedge your bets if you were uncertain.

abruzzi 4y ago

I remember taking a multiple choice question test for some scholarship 35 years ago. 20 questions with the explicit instructions that for any question all of the choices could be correct or none of them or combination, and you were supposed to check only the answers that were correct. It was a VERY hard test. In addition you could score less than zero since every correct item that you checked would add points but every incorrect item itme you checked would subtract points AND every correct item you didn't check would also subtract points. You didn't get points for not checking incorrect items. I've never seen a test like that for school work.

baq 4y ago
I had a few of those in university, they were a favorite of a particular professor for a while. On the very first exam, given that you had to get 50% of points to get a passing grade, IIRC about 20% had a score above 0. Those exams got easier after a couple years. I imagine there were some words to be had with the dean.

Someone 4y ago

I would choose A, too, but if you can do A, the questionnaire isn’t anonymous. I would go for A’: email all of them, reminding them to fill in the questionnaire, if they haven’t already done so.