3 ms·
IMO the theory at the start of the post is well written and almost there, but it needs to more substantively engage with the relevant philosophical concepts. As
by bbor 2y ago
IMO the theory at the start of the post is well written and almost there, but it needs to more substantively engage with the relevant philosophical concepts. As a result, the title "efficiency is bad!" is incorrect in my opinion.
That said, the post is still valuable and would work much better with a framing closer to "some analogies between statistical analysis and public policy" -- the rest of the post (all the political recommendations) is honestly really solid, even if I don't see a lot of the particular examples' connections to their analogous ML approaches. The creativity is impressive, and overall I think it's a productive, thought-provoking exercise. Thanks for posting OP!
Now, for any fellow pendants, the philosophical critique:
more efficient centralized tracking of student progress by standardized testing
The bad part of standardized testing isn't at all that it's "too efficient", it's that it doesn't measure all the educational outcomes we desire. That's just regular ol' flawed metrics.
This same counterintuitive relationship between efficiency and outcome occurs in machine learning, where it is called overfitting.
Again, overfitting isn't an example of a model being too efficacious, much less too efficient (which IMO is, in technical contexts, a measure of speed/resource consumption and not related to accuracy in the first place).
Overfitting on your dataset just means that you built a (virtual/non-actual) model that doesn't express the underlying (virtual) pattern you're concerned with, but rather a subset of that pattern. That's not even a problem necessarily, if you know what subset you've expressed -- words like "under"/"too close" come into play when it's a random or otherwise meaningless subset.
I'm not allowed to train my model on the test dataset though (that would be cheating), so I instead train the model on a proxy dataset, called the training dataset.
I'd say that both the training and test sets are actualized expressions of your targeted virtual pattern. 100% training accuracy means little if it breaks in online, real-world use.
When a measure becomes a target, if it is effectively optimized, then the thing it is designed to measure will grow worse.
I'd take this as proof that what we're really talking about here is efficacy, not efficiency. This is cute and much better than the opening/title, but my critique above tells me that this is just a wordy rephrasing of "different things have differences". That certainly backs up their claim that the proposed law is universal, at least!