4 ms·

What, specifically, is an example of an exam not measuring proficiency? If an exam is well designed, the student will need to figure out what is being asked and

What, specifically, is an example of an exam not measuring proficiency? If an exam is well designed, the student will need to figure out what is being asked and use their mastery to provide an answer.

marksbrown 2y ago
A good example in the UK is teaching students the FOIL technique for algebraic expansion. Students typically can expand (ax+b)(cx+d) because they've learnt a recipe but cannot expand say (ax2+bx+c)(dx+e). Many schools here focus on such tricks (nix the tricks was a great book focusing on such things) as schools here are judged on pass/fail rates. In general, exams are an excellent way to assess students en masse at their ability to remember similar problems but not inherent problem solving techniques. The latter I've found is possible to teach 1to1 but far harder with a class of varying abilities.
- LargeWu 2y ago
  That, to me, is not a problem with the exam though. It's a problem of teaching to a special case and not the general case. If you want to find fault, it's in the incentive system. But I don't see how the exam itself is the problem.
  - marksbrown 2y ago
    Well I won't reiterate all of 'bad education' by Bryan Caplan but to my mind exams are imperfect because: 1. Schools are not equal. It's not fair to compare students when they usually have no choice over their teachers. 2. Exams cover an arbitrary syllabus controlled by undemocratic exam boards. 3. Topics are chosen by exam boards that can be examined not by importance. 4. Students who perform poorly under stress of exam conditions are punished for it. 5. Exams serve no real purpose. Children are not chickens being graded for sale. They're at best a weak signal of aptitude. I would much prefer exams to serve as a prerequisite of sitting a future course rather than an assessment at the end. That way teachers can actually teach rather than continuously repeat the same content.
- pokerface_86 2y ago
  the inability the generalize the foil procedure to an expression with more than 2 variables speaks more to the non mathematically oriented population just sucking at generalizing things. i have found this to be a very “you have it or you don’t” type of thing, not really something that can be taught
  - snowfarthing 2y ago
    Then again, it may be because FOIL is stupid. I've always had a difficult time wrapping my head around this acronym. What counts as "outer"? What counts as "inner"? And yes, when there are more than two items (not necessarily variables!) to be multiplied, you suddenly have to ignore this little trick, because now it's confusing to know what to do about the middle stuff -- and it doesn't take into account non-commutativity either. And yes, some of the problem may be due to my (very recently diagnosed! at least, formally) autistic mind. But I cannot help but think that if someone with a PhD in math struggles with and largely ignores "FOIL", then the problem may be with the technique, and not with the people who don't understand it.
    - pokerface_86 2y ago
      i never found foil stupid, but i also only have a bachelors in math. maybe this was because i had already been exposed to multiplying polynomial expressions, beyond just a 2 term * 2 term by the time i had learned it in school, but i never found it particularly complicated to grasp. foil was never taught to me as the only way to multiply polynomials, rather, an easy algorithm to apply in a certain case. the goal is for you to make the connection that oh, in a 3x2 case, u have to multiply each term in the 3 with each term in the 2, etc. i think your problems with foil can be extended to the general way math is taught. at least for me, it was always full of tricks, little rules that can be broken sometimes, and i was constantly learning new things that made me realize my old teachers had taught us tricks to shortcut solutions.
csa 2y ago
> What, specifically, is an example of an exam not measuring proficiency Not op, but a few examples: 1. Test structures that reward good time management. The paper SAT is a good example of this. The early items in a section are (were?) easy, the middle items were medium difficulty, and the toughest items were towards the end. A good test take would manage their time so that they spent the most amount of time on questions that could improve their scores — later questions for high achievers, but the middle questions for folks who were missing a lot of questions in that range. 2. Test structures that reward endurance. Again, SAT is a good example. How often have examinees sat down in a multi-hour high-stakes testing situation before taking the SAT. “Not often enough to feel comfortable” is usually the answer. I have taken some foreign language proficiency tests that were also multi-hour long endeavors. I practiced answer old exam questions in a testing environment, so the tests were easy for me. Some of my peers did not, and fatigue seemed to be a part of their challenge. 3. I took some “issue spotter” tests in college. I had never heard of or seen an issue spotter test before. I bombed my first one. My professor kindly walked me through things he knew I knew the answers to that would what given me credit. I aced all future issue-spotter exams. Side note: familiarity with test/question format seems to matter for better students, but it largely doesn’t for unmotivated students. Many studies on familiarity with test/question format show no correlation, but my personal experience (as a test giver) is that this is due to lumping the unmotivated and ambivalent examinees with the folks who notice, care, and take action. There are many more answers to your question, but the above are few decent examples.
- Retric 2y ago
  I think test prep allowing people to increase SAT scores is actually a useful feature not a bug. If you’re the kind of person who’s going to put extra effort to add a few points to your results, you are also the kind of person more likely to do well in collage classes.
  - csa 2y ago
    I agree in general with what your idea, but this sort of assumes a few things that aren’t always true: 1. Examinee is aware of test prep and its potential benefits. Definitely not always true in low socioeconomic status (SES) families, especially ones with no family members who have been to college. 2. Examinee can afford said test prep courses and/or materials. Library and online are “free” options, but we are back to assuming that the examinee has easy access to a library and/or the internet and knows how to find said materials. There are many, many students who would do well in college if they had some insight on how to do college. The US has a lot of wasted potential due to our public school focus on bringing up the low achievers to the exclusion of developing those with high achiever potential. Some private schools and most public schools in “good” neighborhoods have programs and cultures that cater to those with high potential, but these schools only address a relatively small portion of the student population with high potential, imho.
Retric 2y ago
“Well designed? is doing heavy lifting here. You can get very good at specific test formats in terms of time management, common tricks, etc. Actual tests include things like: Multiple choice questions were providing answers aids answering the question. Short responses / fill in the blank generally mean people can just regurgitate answers they don’t understand. Essay responses sound great, except you can’t answer many questions and essays writing is a separate skill which heavily influences final scores. More broadly tests are time limited so can’t test skills that take long periods of time to demonstrate a major issue for say programming.
- margalabargala 2y ago
  > More broadly tests are time limited so can’t test skills that take long periods of time to demonstrate a major issue for say programming. Every programming class I've taken, in high school or college, was project based where the main source of grade were actual programs I wrote which actually did something. The one exception perhaps being the AP test for AP CS.
brainwipe 2y ago
Exams are a poor measure of proficiency. Proficiency is gained by doing and stretching a skill over time. You can measure that in small increments than a periodic exam. At the end of a period, a student would have a body of work to demonstrate proficiency rather than relying on a single day. When I taught at university there was a disparity between exam grades and the physical body of work they had submitted over the years. You'd see the grade and be shocked the did so badly. The grade reflected proficiency in examinations, not in the subject.
screcth 2y ago
English exams for ESL students are a great example. Getting good grades in those exams requires that you know the criteria for evaluating each part of the exam and how to tailor your answer to that criteria. For example, if the exam asks you to write a short movie review you are expected to follow the formula for reviews and show that you can use certain specific grammar constructs. If you know English well but you don't practice the exam before you will get a mediocre grade, simply because you didn't follow the tacit guidelines that you are expected to know.

marksbrown 2y ago

A good example in the UK is teaching students the FOIL technique for algebraic expansion. Students typically can expand (ax+b)(cx+d) because they've learnt a recipe but cannot expand say (ax2+bx+c)(dx+e). Many schools here focus on such tricks (nix the tricks was a great book focusing on such things) as schools here are judged on pass/fail rates. In general, exams are an excellent way to assess students en masse at their ability to remember similar problems but not inherent problem solving techniques. The latter I've found is possible to teach 1to1 but far harder with a class of varying abilities.

LargeWu 2y ago
That, to me, is not a problem with the exam though. It's a problem of teaching to a special case and not the general case. If you want to find fault, it's in the incentive system. But I don't see how the exam itself is the problem.
- marksbrown 2y ago
  Well I won't reiterate all of 'bad education' by Bryan Caplan but to my mind exams are imperfect because: 1. Schools are not equal. It's not fair to compare students when they usually have no choice over their teachers. 2. Exams cover an arbitrary syllabus controlled by undemocratic exam boards. 3. Topics are chosen by exam boards that can be examined not by importance. 4. Students who perform poorly under stress of exam conditions are punished for it. 5. Exams serve no real purpose. Children are not chickens being graded for sale. They're at best a weak signal of aptitude. I would much prefer exams to serve as a prerequisite of sitting a future course rather than an assessment at the end. That way teachers can actually teach rather than continuously repeat the same content.
pokerface_86 2y ago
the inability the generalize the foil procedure to an expression with more than 2 variables speaks more to the non mathematically oriented population just sucking at generalizing things. i have found this to be a very “you have it or you don’t” type of thing, not really something that can be taught
- snowfarthing 2y ago
  Then again, it may be because FOIL is stupid. I've always had a difficult time wrapping my head around this acronym. What counts as "outer"? What counts as "inner"? And yes, when there are more than two items (not necessarily variables!) to be multiplied, you suddenly have to ignore this little trick, because now it's confusing to know what to do about the middle stuff -- and it doesn't take into account non-commutativity either. And yes, some of the problem may be due to my (very recently diagnosed! at least, formally) autistic mind. But I cannot help but think that if someone with a PhD in math struggles with and largely ignores "FOIL", then the problem may be with the technique, and not with the people who don't understand it.
  - pokerface_86 2y ago
    i never found foil stupid, but i also only have a bachelors in math. maybe this was because i had already been exposed to multiplying polynomial expressions, beyond just a 2 term * 2 term by the time i had learned it in school, but i never found it particularly complicated to grasp. foil was never taught to me as the only way to multiply polynomials, rather, an easy algorithm to apply in a certain case. the goal is for you to make the connection that oh, in a 3x2 case, u have to multiply each term in the 3 with each term in the 2, etc. i think your problems with foil can be extended to the general way math is taught. at least for me, it was always full of tricks, little rules that can be broken sometimes, and i was constantly learning new things that made me realize my old teachers had taught us tricks to shortcut solutions.

csa 2y ago

> What, specifically, is an example of an exam not measuring proficiency Not op, but a few examples: 1. Test structures that reward good time management. The paper SAT is a good example of this. The early items in a section are (were?) easy, the middle items were medium difficulty, and the toughest items were towards the end. A good test take would manage their time so that they spent the most amount of time on questions that could improve their scores — later questions for high achievers, but the middle questions for folks who were missing a lot of questions in that range. 2. Test structures that reward endurance. Again, SAT is a good example. How often have examinees sat down in a multi-hour high-stakes testing situation before taking the SAT. “Not often enough to feel comfortable” is usually the answer. I have taken some foreign language proficiency tests that were also multi-hour long endeavors. I practiced answer old exam questions in a testing environment, so the tests were easy for me. Some of my peers did not, and fatigue seemed to be a part of their challenge. 3. I took some “issue spotter” tests in college. I had never heard of or seen an issue spotter test before. I bombed my first one. My professor kindly walked me through things he knew I knew the answers to that would what given me credit. I aced all future issue-spotter exams. Side note: familiarity with test/question format seems to matter for better students, but it largely doesn’t for unmotivated students. Many studies on familiarity with test/question format show no correlation, but my personal experience (as a test giver) is that this is due to lumping the unmotivated and ambivalent examinees with the folks who notice, care, and take action. There are many more answers to your question, but the above are few decent examples.

Retric 2y ago
I think test prep allowing people to increase SAT scores is actually a useful feature not a bug. If you’re the kind of person who’s going to put extra effort to add a few points to your results, you are also the kind of person more likely to do well in collage classes.
- csa 2y ago
  I agree in general with what your idea, but this sort of assumes a few things that aren’t always true: 1. Examinee is aware of test prep and its potential benefits. Definitely not always true in low socioeconomic status (SES) families, especially ones with no family members who have been to college. 2. Examinee can afford said test prep courses and/or materials. Library and online are “free” options, but we are back to assuming that the examinee has easy access to a library and/or the internet and knows how to find said materials. There are many, many students who would do well in college if they had some insight on how to do college. The US has a lot of wasted potential due to our public school focus on bringing up the low achievers to the exclusion of developing those with high achiever potential. Some private schools and most public schools in “good” neighborhoods have programs and cultures that cater to those with high potential, but these schools only address a relatively small portion of the student population with high potential, imho.

Retric 2y ago

“Well designed? is doing heavy lifting here. You can get very good at specific test formats in terms of time management, common tricks, etc. Actual tests include things like: Multiple choice questions were providing answers aids answering the question. Short responses / fill in the blank generally mean people can just regurgitate answers they don’t understand. Essay responses sound great, except you can’t answer many questions and essays writing is a separate skill which heavily influences final scores. More broadly tests are time limited so can’t test skills that take long periods of time to demonstrate a major issue for say programming.

margalabargala 2y ago
> More broadly tests are time limited so can’t test skills that take long periods of time to demonstrate a major issue for say programming. Every programming class I've taken, in high school or college, was project based where the main source of grade were actual programs I wrote which actually did something. The one exception perhaps being the AP test for AP CS.

brainwipe 2y ago

Exams are a poor measure of proficiency. Proficiency is gained by doing and stretching a skill over time. You can measure that in small increments than a periodic exam. At the end of a period, a student would have a body of work to demonstrate proficiency rather than relying on a single day. When I taught at university there was a disparity between exam grades and the physical body of work they had submitted over the years. You'd see the grade and be shocked the did so badly. The grade reflected proficiency in examinations, not in the subject.

screcth 2y ago

English exams for ESL students are a great example. Getting good grades in those exams requires that you know the criteria for evaluating each part of the exam and how to tailor your answer to that criteria. For example, if the exam asks you to write a short movie review you are expected to follow the formula for reviews and show that you can use certain specific grammar constructs. If you know English well but you don't practice the exam before you will get a mediocre grade, simply because you didn't follow the tacit guidelines that you are expected to know.