3 ms·

This problem isn't a teaching problem. Evaluating someone's skills in any respect or context is basically an intractable problem. Interviewing, school, job perf

This problem isn't a teaching problem. Evaluating someone's skills in any respect or context is basically an intractable problem. Interviewing, school, job performance, etc. etc. If there were an organization that could "perfectly" evaluate people's skills in a fixed period of time it would quickly become the top, and eventually only company. It would use its own skills in order to remove low performers, perfectly from its own organization. It would find all of the top performers outside of the organization, perfecting arbitrating wage vs. value benefits. Profits from this would be divested back into the organization forming an infinite virtuous cycle. Later it would supersede whatever nation it's in, conquering it by finding the best military leaders and soldiers using the same "perfect evaluation" ability. It would get the best diplomats and business leaders. Later it would turn an eye to other nations, then the world. Eventually the galaxy and the entire universe.

thrill 5y ago
We could call it the Paperclip Maximizer.
akvadrako 5y ago
There is such a system in the long run, it's called the free market. If you can consistently outperform your peers while both parties have complete information, it's a sign of having some advantage. Free markets are never perfect locally, but on a galactic scale they are pretty close, so the superior groups and will dominate.
- brabel 5y ago
  > so the superior groups and will dominate. That may be true by definition, if your definition is that superior groups eventually dominate, but that's of course just tautology. However, depending on how you define "superior", for example "more intelligent and honest", or "more compassionate and fairer", could be what most people have in mind, then that may not be true at all. In human societies, throughout history, it's likely that who dominates is actually the most brutal and reckless, up to a point where people actually become accountable for their actions.
- chaosite 5y ago
  See, no, the second you put in "superior" then you left any idea of a free market. The idea of a free market doesn't claim to make any value judgement of what group is better, the free market is purely about selecting fair prices for commodities. Your idea of superior groups and so on based on success on the free market is basically social darwinism.
- DannyBee 5y ago
  Actually, free markets are only efficient if p=np. (They are not strong form efficient, that was disproven long ago, and are only weak form efficient if p=np) Unfortunately, this hasn't stopped people from believing in them anyway, because they really really want them to work, and people "feel" like they should
  - mlyle 5y ago
    They still outperform other mechanisms we have of resource allocation. They're massively parallel systems with ample signaling of different agents' state. And the worst corner cases of externalities can be mostly offset with proper regulation, even if we struggle to do it.
matthewdgreen 5y ago
The first thing to understand is that the goal in teaching is not to evaluate anyone's skills. The goal in teaching is to make sure that students learn things. From a teacher's perspective, the evaluation part is entirely a hack to make sure that they do.
- jessriedel 5y ago
  > The first thing to understand is that the goal in teaching is not to evaluate anyone's skills. The goal in teaching is to make sure that students learn things. No, both learning and certification of learning (in a way legible to 3rd parties) are real and proper goals of teaching.
  - skybrian 5y ago
    They're both important, but I think having exams given by a separate person or organization might be a win in some cases? It might mean more teaching to the test, though.
- asdfasgasdgasdg 5y ago
  I don't think it's true that in all contexts the role of a teacher is exclusively to teach. "Teachers" are also part of a credentialing system used in our society to identify people who are skilled or talented. This is discussed in the article when the author talks about the diffuse harms inflicted by cheaters.
- endisneigh 5y ago
  I disagree - without evaluation how do they “make sure that students learn things”?
- arendtio 5y ago
  I think this is a crucial point missing in the above discussion. However, tests should have a place in the teachers perspective, because they improve the learning effect (because they trigger the memory retrieval reliably). So maybe the problem is simply, that tests are linked to grades ;-)
jessriedel 5y ago
> This problem isn't a teaching problem. Evaluating someone's skills in any respect or context is basically an intractable problem. You're not engaging with author's argument. The author explicitly assumes for the sake of argument that perfect evaluation is possible. He's saying that even under this unrealistic assumption, teacher policies that naively look draconian are in fact hard to avoid given reasonable teacher effort.
- endisneigh 5y ago
  That wasn’t my read. Ultimately the issues described are political. The author says as much when describing the “structural forces” and that systems with humans behave is funny ways.
  - jessriedel 5y ago
    That two things both involve politics does not, by itself, imply they cannot be usefully discussed separately. You need to actually argue that the issues the author discusses go away or become moot given the imperfection of real-world evaluation. It is not enough to argue that the world would look different were perfect evaluation possible.
hajile 5y ago
In ancient times, all final exams would be oral ones in front of a panel of teachers. I'd guess that this technique would be pretty successful today too.
- titanomachy 5y ago
  This works great, and it is still how evaluation is done when the stakes are higher: PhD defenses, executive hiring... even getting hired as an entry-level engineer at Google requires about five hours of what is basically oral examination. But society is not willing to pay that kind of price for the earlier levels of evaluation. We want "scalable" systems. Unfortunately those same evaluations are often treated as more sensitive than they really are. For example, if you're comparing two students I'd argue that 3.0 vs 3.5 GPA gives you at least some signal, whereas 3.5 vs 3.6 GPA gives you basically no signal at all (maybe the 3.6 student took easier courses, maybe they were more lucky with cutoffs, etc.). And yet the distinction sometimes matters e.g. to graduate programs. In well-designed systems, the GPA cutoff is set relatively low and more sensitive methods are used to select the best students from the pool. Often this includes an interview with a professor, which is also a form of oral exam.
  - hajile 5y ago
    You don't need oral arguments throughout the process. One at the end of the course is sufficient. It is then up to the students to learn the requisite knowledge in the given time period.

thrill 5y ago

We could call it the Paperclip Maximizer.

akvadrako 5y ago

There is such a system in the long run, it's called the free market. If you can consistently outperform your peers while both parties have complete information, it's a sign of having some advantage. Free markets are never perfect locally, but on a galactic scale they are pretty close, so the superior groups and will dominate.

brabel 5y ago
> so the superior groups and will dominate. That may be true by definition, if your definition is that superior groups eventually dominate, but that's of course just tautology. However, depending on how you define "superior", for example "more intelligent and honest", or "more compassionate and fairer", could be what most people have in mind, then that may not be true at all. In human societies, throughout history, it's likely that who dominates is actually the most brutal and reckless, up to a point where people actually become accountable for their actions.
chaosite 5y ago
See, no, the second you put in "superior" then you left any idea of a free market. The idea of a free market doesn't claim to make any value judgement of what group is better, the free market is purely about selecting fair prices for commodities. Your idea of superior groups and so on based on success on the free market is basically social darwinism.
DannyBee 5y ago
Actually, free markets are only efficient if p=np. (They are not strong form efficient, that was disproven long ago, and are only weak form efficient if p=np) Unfortunately, this hasn't stopped people from believing in them anyway, because they really really want them to work, and people "feel" like they should
- mlyle 5y ago
  They still outperform other mechanisms we have of resource allocation. They're massively parallel systems with ample signaling of different agents' state. And the worst corner cases of externalities can be mostly offset with proper regulation, even if we struggle to do it.

matthewdgreen 5y ago

The first thing to understand is that the goal in teaching is not to evaluate anyone's skills. The goal in teaching is to make sure that students learn things. From a teacher's perspective, the evaluation part is entirely a hack to make sure that they do.

jessriedel 5y ago
> The first thing to understand is that the goal in teaching is not to evaluate anyone's skills. The goal in teaching is to make sure that students learn things. No, both learning and certification of learning (in a way legible to 3rd parties) are real and proper goals of teaching.
- skybrian 5y ago
  They're both important, but I think having exams given by a separate person or organization might be a win in some cases? It might mean more teaching to the test, though.
asdfasgasdgasdg 5y ago
I don't think it's true that in all contexts the role of a teacher is exclusively to teach. "Teachers" are also part of a credentialing system used in our society to identify people who are skilled or talented. This is discussed in the article when the author talks about the diffuse harms inflicted by cheaters.
endisneigh 5y ago
I disagree - without evaluation how do they “make sure that students learn things”?
arendtio 5y ago
I think this is a crucial point missing in the above discussion. However, tests should have a place in the teachers perspective, because they improve the learning effect (because they trigger the memory retrieval reliably). So maybe the problem is simply, that tests are linked to grades ;-)

jessriedel 5y ago

> This problem isn't a teaching problem. Evaluating someone's skills in any respect or context is basically an intractable problem. You're not engaging with author's argument. The author explicitly assumes for the sake of argument that perfect evaluation is possible. He's saying that even under this unrealistic assumption, teacher policies that naively look draconian are in fact hard to avoid given reasonable teacher effort.

endisneigh 5y ago
That wasn’t my read. Ultimately the issues described are political. The author says as much when describing the “structural forces” and that systems with humans behave is funny ways.
- jessriedel 5y ago
  That two things both involve politics does not, by itself, imply they cannot be usefully discussed separately. You need to actually argue that the issues the author discusses go away or become moot given the imperfection of real-world evaluation. It is not enough to argue that the world would look different were perfect evaluation possible.

hajile 5y ago

In ancient times, all final exams would be oral ones in front of a panel of teachers. I'd guess that this technique would be pretty successful today too.

titanomachy 5y ago
This works great, and it is still how evaluation is done when the stakes are higher: PhD defenses, executive hiring... even getting hired as an entry-level engineer at Google requires about five hours of what is basically oral examination. But society is not willing to pay that kind of price for the earlier levels of evaluation. We want "scalable" systems. Unfortunately those same evaluations are often treated as more sensitive than they really are. For example, if you're comparing two students I'd argue that 3.0 vs 3.5 GPA gives you at least some signal, whereas 3.5 vs 3.6 GPA gives you basically no signal at all (maybe the 3.6 student took easier courses, maybe they were more lucky with cutoffs, etc.). And yet the distinction sometimes matters e.g. to graduate programs. In well-designed systems, the GPA cutoff is set relatively low and more sensitive methods are used to select the best students from the pool. Often this includes an interview with a professor, which is also a form of oral exam.
- hajile 5y ago
  You don't need oral arguments throughout the process. One at the end of the course is sufficient. It is then up to the students to learn the requisite knowledge in the given time period.