3 ms·

Isn't it a reasonable filter? I'd (naively) assume that all things being equal, a person who passes a test on the first attempt, is more likely to have a higher

Isn't it a reasonable filter? I'd (naively) assume that all things being equal, a person who passes a test on the first attempt, is more likely to have a higher innate ability than one who doesn't. Is this wrong?

UltraSane 2y ago
Imagine if they only let people take the bar exam once.
- deleted 2y ago
  [deleted]
michaelt 2y ago
That depends on the test, and the training surrounding it. If I'm hiring for a job where there's a strict requirement to be 6 feet tall and there's nothing candidates can do to get taller - then once you've measured under 6 feet there's no point in re-testing you. If I'm hiring for a job where there's a strict requirement that you be able to lift a 90 lbs weight, a lot of people will fail - but if they hit the gym, in a month or two they'll be able to pass easily. If I'm hiring for a job where there's a strict requirement that you be able to lift a 90 lbs weight, but before the test every applicant goes through a 6 month strength training program, so we test them at absolute peak strength? Re-testing would be a waste of time, except in a few cases like if someone had a loved one die just before the final test. If I'm hiring for a job with a written test, the test draws a random selection of questions from a question bank, and some questions are much easier than others? There might be a substantial random element to the results. If re-testing people produces substantially different scores, probably that indicates I need to improve the design of my test.
cutemonster 2y ago
Yes, wrong I think. People's performance varies from day to day, and if doing the test just once, then some people will pass it because of good luck shape that day, although another day they would have failed. When constantly predictable high performance is important, it makes sense to do the test on a bunch of different occasions, and look at the overall result (like, lowest, median and average scores). And it'd be ok to do a bit worse on one occasion, as long as you were always above the minimum. Rather than in effect randomly picking just one test result. Looking at the results from many tests, reduces false negatives, but also false positives. (And of course if studying and practice matters, getting to re-take the test some years later: yes of course, but possibly the test should then be a little bit longer.)

UltraSane 2y ago

Imagine if they only let people take the bar exam once.

deleted 2y ago
[deleted]

michaelt 2y ago

That depends on the test, and the training surrounding it. If I'm hiring for a job where there's a strict requirement to be 6 feet tall and there's nothing candidates can do to get taller - then once you've measured under 6 feet there's no point in re-testing you. If I'm hiring for a job where there's a strict requirement that you be able to lift a 90 lbs weight, a lot of people will fail - but if they hit the gym, in a month or two they'll be able to pass easily. If I'm hiring for a job where there's a strict requirement that you be able to lift a 90 lbs weight, but before the test every applicant goes through a 6 month strength training program, so we test them at absolute peak strength? Re-testing would be a waste of time, except in a few cases like if someone had a loved one die just before the final test. If I'm hiring for a job with a written test, the test draws a random selection of questions from a question bank, and some questions are much easier than others? There might be a substantial random element to the results. If re-testing people produces substantially different scores, probably that indicates I need to improve the design of my test.

cutemonster 2y ago

Yes, wrong I think. People's performance varies from day to day, and if doing the test just once, then some people will pass it because of good luck shape that day, although another day they would have failed. When constantly predictable high performance is important, it makes sense to do the test on a bunch of different occasions, and look at the overall result (like, lowest, median and average scores). And it'd be ok to do a bit worse on one occasion, as long as you were always above the minimum. Rather than in effect randomly picking just one test result. Looking at the results from many tests, reduces false negatives, but also false positives. (And of course if studying and practice matters, getting to re-take the test some years later: yes of course, but possibly the test should then be a little bit longer.)