4 ms·

All true, of course. But numerically accurate data on a proxy quantity that you don't care about are not necessarily inferior to numerically inaccurate data on

by lutorm 14y ago

All true, of course. But numerically accurate data on a proxy quantity that you don't care about are not necessarily inferior to numerically inaccurate data on the quantity you care about. Both pose challenges to interpretation. It's easy to be seduced by clear numbers and worry less about what the numbers mean. We can argue about which fields count as "science", but many fields try to learn from non-reproducible data. (And I say this having spent my career in the physical sciences. There's too much noise induced by people who present statistically solid analyses of numbers whose connection to the core questions is much more tenuous...)

yummyfajitas 14y ago
Standardized tests typically measure how good a student is at adding, subtracting, and reading a paragraph and using the information it contains. Why do you feel we don't care about that? What information should we care about?
- dsanderson 14y ago
  Standardized tests claim to measure those quantities. To be fair, my school system was quite good, so I don't think I suffered too greatly from teaching to the test, but I did read quite a few of the study/test guides put out by Kaplan and their ilk. Much of it was tricks based on the structure of the tests; the "types" of question that will be asked and type-specific solution strategies. These tests are not about basic understanding and skill at adding, reading comprehension, etc., but rather a well-defined subset of those skills. The assumption is that skill in the subset is a good proxy for fundamental skills in the student. If teachers are evaluated solely by these partial measures, it is in their best interest (and, in the eyes of the administrators, produces the best results) if they optimize for these partial measures; the argument from standardized testing opponents is that teaching for these partial measures has not been shown to produce general understanding. In that sense, NCLB created a race to the bottom; schools that performed below-average got less funding, so if any schools successfully "taught to the test", they all had too or face budget cuts (that's my understanding, IANAT). Is there evidence that these partial measures reveal and/or necessarily lead to a more general understanding?
  - yummyfajitas 14y ago
    Here is an actual CA standardized test: http://www.cde.ca.gov/ta/tg/sr/documents/cstrtqmath7.pdf http://www.cde.ca.gov/ta/tg/sr/documents/cstrtqmath7.pdf Please explain why you feel it doesn't adequately measure the ability of students to "read, write, and compare rational numbers in scientific notation (positive and negative powers of 10) with approximate numbers using scientific notation" (as well as the other topics listed). Or, similarly, can you explain specifically what "teaching to the test" means and why it is suboptimal for learning the aforementoined topics?
    - dsanderson 14y ago
      Sorry for the delay, I didn't realize you had replied (is there some global notification UI element I should look for?) I don't know if you'll see this, but I'll reply anyway. I'm not from California, nor have I studied their tests, nor did I claim that standardized tests themselves are flawed, so your first demand is misleading, at best. I will grant that, having done further research, my first examples of teaching to the test were flawed; I'll try to summarize it (and my position) below. Teaching for the test is explained pretty well here: http://www.carnegiefoundation.org/perspectives/teaching-test http://www.carnegiefoundation.org/perspectives/teaching-test and on wikipedia [ http://en.wikipedia.org/wiki/Teaching_to_the_test http://en.wikipedia.org/wiki/Teaching_to_the_test ]. Note that this definition of teaching to the test isn't necessarily bad, only when applied wrong. Punishing/rewarding schools based on student performance on a small set of tests creates an incentive to have student "learn" by mercilessly practicing on those tests, to the exclusion of other approaches. As per the sources on wikipedia, this doesn't impart a "general understanding," nor does it necessarily raise test scores. Other people here have linked stories where teachers and schools actually helped their students cheat on standardized tests. Standardized tests themselves aren't the problem; we've had them for decades, and they can provide valuable insights in some cases. The problem is when we focus on standardized tests to the exclusion of all else, putting a perverse pressure on schools. I believe you previously supported evaluating by standardized tests because they're the only quantitative measure we have; maybe this means we can't reliably quantitatively evaluate the impact of a single teacher on a small set of heterogeneous subjects. Demonstrate we can, then I'll reconsider NCLB. edit to add another thought: I'd liken this to the problem of overfitting in machine learning. You have a way of measuring the performance of your system, and try to improve the system via that feedback. It's easy to overfit it, making it work beautifully on the provided data, but lack generality. The problem isn't in how we measure "fitness," necessarily, it is in how we update the system based on that result. That is, standardized tests may be a fine measure among students with that "generalized understanding"; I've not been convinced we are using those results intelligently.

by lutorm 14y ago

yummyfajitas 14y ago

Standardized tests typically measure how good a student is at adding, subtracting, and reading a paragraph and using the information it contains. Why do you feel we don't care about that? What information should we care about?

dsanderson 14y ago
Standardized tests claim to measure those quantities. To be fair, my school system was quite good, so I don't think I suffered too greatly from teaching to the test, but I did read quite a few of the study/test guides put out by Kaplan and their ilk. Much of it was tricks based on the structure of the tests; the "types" of question that will be asked and type-specific solution strategies. These tests are not about basic understanding and skill at adding, reading comprehension, etc., but rather a well-defined subset of those skills. The assumption is that skill in the subset is a good proxy for fundamental skills in the student. If teachers are evaluated solely by these partial measures, it is in their best interest (and, in the eyes of the administrators, produces the best results) if they optimize for these partial measures; the argument from standardized testing opponents is that teaching for these partial measures has not been shown to produce general understanding. In that sense, NCLB created a race to the bottom; schools that performed below-average got less funding, so if any schools successfully "taught to the test", they all had too or face budget cuts (that's my understanding, IANAT). Is there evidence that these partial measures reveal and/or necessarily lead to a more general understanding?
- yummyfajitas 14y ago
  Here is an actual CA standardized test: http://www.cde.ca.gov/ta/tg/sr/documents/cstrtqmath7.pdf http://www.cde.ca.gov/ta/tg/sr/documents/cstrtqmath7.pdf Please explain why you feel it doesn't adequately measure the ability of students to "read, write, and compare rational numbers in scientific notation (positive and negative powers of 10) with approximate numbers using scientific notation" (as well as the other topics listed). Or, similarly, can you explain specifically what "teaching to the test" means and why it is suboptimal for learning the aforementoined topics?
  - dsanderson 14y ago
    Sorry for the delay, I didn't realize you had replied (is there some global notification UI element I should look for?) I don't know if you'll see this, but I'll reply anyway. I'm not from California, nor have I studied their tests, nor did I claim that standardized tests themselves are flawed, so your first demand is misleading, at best. I will grant that, having done further research, my first examples of teaching to the test were flawed; I'll try to summarize it (and my position) below. Teaching for the test is explained pretty well here: http://www.carnegiefoundation.org/perspectives/teaching-test http://www.carnegiefoundation.org/perspectives/teaching-test and on wikipedia [ http://en.wikipedia.org/wiki/Teaching_to_the_test http://en.wikipedia.org/wiki/Teaching_to_the_test ]. Note that this definition of teaching to the test isn't necessarily bad, only when applied wrong. Punishing/rewarding schools based on student performance on a small set of tests creates an incentive to have student "learn" by mercilessly practicing on those tests, to the exclusion of other approaches. As per the sources on wikipedia, this doesn't impart a "general understanding," nor does it necessarily raise test scores. Other people here have linked stories where teachers and schools actually helped their students cheat on standardized tests. Standardized tests themselves aren't the problem; we've had them for decades, and they can provide valuable insights in some cases. The problem is when we focus on standardized tests to the exclusion of all else, putting a perverse pressure on schools. I believe you previously supported evaluating by standardized tests because they're the only quantitative measure we have; maybe this means we can't reliably quantitatively evaluate the impact of a single teacher on a small set of heterogeneous subjects. Demonstrate we can, then I'll reconsider NCLB. edit to add another thought: I'd liken this to the problem of overfitting in machine learning. You have a way of measuring the performance of your system, and try to improve the system via that feedback. It's easy to overfit it, making it work beautifully on the provided data, but lack generality. The problem isn't in how we measure "fitness," necessarily, it is in how we update the system based on that result. That is, standardized tests may be a fine measure among students with that "generalized understanding"; I've not been convinced we are using those results intelligently.