5 ms·

Shorter is good, but dynamic tests (called "computer adaptive tests" or CAT in the industry) make it difficult to follow longstanding best-practices for test ta

by glial 3y ago

Shorter is good, but dynamic tests (called "computer adaptive tests" or CAT in the industry) make it difficult to follow longstanding best-practices for test taking. When I took the SAT, I could quickly scan through the questions in the current section and mark those that looked difficult, and save them for the end. With a CAT test, it's typically one-question-at-a-time, and if you're a strong test taker, they get successively more difficult. They are more efficient tests, but (IMHO) less pleasant to take.

gnicholas 3y ago
That is true for adaptive tests that operate on a per-question basis. What I have read of the new SAT is that it operates at a session level. So if you do well on the entire first session, you get a harder version of the next session on that topic (reading or math). Students should still be able to use the technique you mention to identify hard questions and save them for last.
- Kon-Peki 3y ago
  > What I have read of the new SAT is that it operates at a session level. So if you do well on the entire first session, you get a harder version of the next session on that topic (reading or math). That is positive. I've seen issues with adaptive tests not being resilient to accidents and brain farts: if a kid fat-fingers the response to a question early in the test, they may never get the computer to give them questions of a suitable difficulty later in the test. Thus they never even have a chance to get a score that accurately reflects their ability. If the SAT is session-based, each session needs to be large enough that a single question can't tank the whole thing, but at the same time there also needs to be enough sessions to allow for properly dialing-in the knowledge level. In reality, the ideal situation is that the test should allow for an unlimited number of sessions, but each session is time-limited. Something along the lines of stopping once the test-taker gets below a certain score on 2 consecutive sessions.
  - gnicholas 3y ago
    > If the SAT is session-based, each session needs to be large enough that a single question can't tank the whole thing I think each session is either half or a third of the test, for that subject. There could actually be more than two versions of the later session, to ensure that a student who makes a mistake or two isn't prevented from getting a relatively high score.
dylan604 3y ago
Some of the math competitions I participated would penalize for skipping around. You were awarded +5 for correct answers and -4 for incorrect or skipped questions. It was not uncommon for negative scores. In one test, any stray marks were also marked as incorrect. No erasing, no changing a 7 to a 9, or anything was allowed. The questions leaned toward an increasing difficulty, but there could something very difficult followed by a string of much easier questions. So additional math had to be done to see if it would be better to stop or skip.
- saagarjha 3y ago
  I don’t see how this is any different from a normal test? If you skip a question or get it wrong the penalty is you don’t get any points for it.
  - dylan604 3y ago
    As stated: "You were awarded +5 for correct answers and -4 for incorrect or skipped questions." Answer question 1 correctly: +5 Skip question 2: -4 Answer question 3 correctly: +5 Net score: +6 If skipped were just ignored, the net score would be +10.
    - saagarjha 3y ago
      Interesting, I don't think I've ever seen a test like that. What was it?
      - dylan604 3y ago
        Again, as stated: "Some of the math competitions" Don't know how wide spread they are in other states, but it was part of the UIL Academics teams as statewide competition between school districts in Texas.
    - tzs 3y ago
      It's still equivalent to "normal" tests, because the cost of skipping and the cost of answering incorrectly are the same. A test in that format with N questions and a test in the normal "1 point for a correct answer, 0 for anything else" with N questions are related as follows, assuming C correct answers: Score in the +5/-4 system: 5 C - 4 (N-C) = 9 C - 4 N Score in the +1/0 system: 1 C + 0 It's basically a "9 points for a correct answer, 0 for anything else" test that simply starts with a score of -4 N. It's very similar to the difference between Celsius and Fahrenheit.
jessriedel 3y ago
> make it difficult to follow longstanding best-practices for test taking. Isn't that good? The effectiveness of test-taking strategies is a (minor) flaw in the SAT. People taking an ideal test would not benefit at all from learning strategies. (The flaw is minor because all our methods for assessing people are gameable to some extent, and standardized general tests like the SAT are among the least gameable.)

by glial 3y ago

gnicholas 3y ago

That is true for adaptive tests that operate on a per-question basis. What I have read of the new SAT is that it operates at a session level. So if you do well on the entire first session, you get a harder version of the next session on that topic (reading or math). Students should still be able to use the technique you mention to identify hard questions and save them for last.

Kon-Peki 3y ago
> What I have read of the new SAT is that it operates at a session level. So if you do well on the entire first session, you get a harder version of the next session on that topic (reading or math). That is positive. I've seen issues with adaptive tests not being resilient to accidents and brain farts: if a kid fat-fingers the response to a question early in the test, they may never get the computer to give them questions of a suitable difficulty later in the test. Thus they never even have a chance to get a score that accurately reflects their ability. If the SAT is session-based, each session needs to be large enough that a single question can't tank the whole thing, but at the same time there also needs to be enough sessions to allow for properly dialing-in the knowledge level. In reality, the ideal situation is that the test should allow for an unlimited number of sessions, but each session is time-limited. Something along the lines of stopping once the test-taker gets below a certain score on 2 consecutive sessions.
- gnicholas 3y ago
  > If the SAT is session-based, each session needs to be large enough that a single question can't tank the whole thing I think each session is either half or a third of the test, for that subject. There could actually be more than two versions of the later session, to ensure that a student who makes a mistake or two isn't prevented from getting a relatively high score.

dylan604 3y ago

Some of the math competitions I participated would penalize for skipping around. You were awarded +5 for correct answers and -4 for incorrect or skipped questions. It was not uncommon for negative scores. In one test, any stray marks were also marked as incorrect. No erasing, no changing a 7 to a 9, or anything was allowed. The questions leaned toward an increasing difficulty, but there could something very difficult followed by a string of much easier questions. So additional math had to be done to see if it would be better to stop or skip.

saagarjha 3y ago
I don’t see how this is any different from a normal test? If you skip a question or get it wrong the penalty is you don’t get any points for it.
- dylan604 3y ago
  As stated: "You were awarded +5 for correct answers and -4 for incorrect or skipped questions." Answer question 1 correctly: +5 Skip question 2: -4 Answer question 3 correctly: +5 Net score: +6 If skipped were just ignored, the net score would be +10.
  - saagarjha 3y ago
    Interesting, I don't think I've ever seen a test like that. What was it?
    - dylan604 3y ago
      Again, as stated: "Some of the math competitions" Don't know how wide spread they are in other states, but it was part of the UIL Academics teams as statewide competition between school districts in Texas.
  - tzs 3y ago
    It's still equivalent to "normal" tests, because the cost of skipping and the cost of answering incorrectly are the same. A test in that format with N questions and a test in the normal "1 point for a correct answer, 0 for anything else" with N questions are related as follows, assuming C correct answers: Score in the +5/-4 system: 5 C - 4 (N-C) = 9 C - 4 N Score in the +1/0 system: 1 C + 0 It's basically a "9 points for a correct answer, 0 for anything else" test that simply starts with a score of -4 N. It's very similar to the difference between Celsius and Fahrenheit.

jessriedel 3y ago

> make it difficult to follow longstanding best-practices for test taking. Isn't that good? The effectiveness of test-taking strategies is a (minor) flaw in the SAT. People taking an ideal test would not benefit at all from learning strategies. (The flaw is minor because all our methods for assessing people are gameable to some extent, and standardized general tests like the SAT are among the least gameable.)