4 ms·

The best method for assessing performance when learning is as old as the world: assess the effort, not how well the result complies with some requirements. If

The best method for assessing performance when learning is as old as the world: assess the effort, not how well the result complies with some requirements. If the level of effort made is high, but the outcome does not comply in some way, praise is due. If the outcome complies, but the level of effort is low, there is no reason for praise (what are you praising? mere compliance?) and you must have set a wrong bar. Not doing this fosters people with mental issues such as rejection anxiety, perfectionism, narcissism, defeatism, etc. If you got good grades at school with little actual effort and the constant praise for that formed your identity, you may be in for a bad time in adulthood. Teacher’s job is to determine the appropriate bar, estimate the level of effort, and to help shape the effort applied in a way that it improves the skill in question and the more general meta skill of learning. The issue of judging by the outcome is prevalent in some (or all) school systems, so we can say LLMs are mostly orthogonal to that. However, even if that issue was addressed, in a number of skills the mere availability of ML-based generative tools makes it impossible to estimate the level of actual effort and to set the appropriate bar, and I do not see how it can be worked around. It’s yet another negative consequence of making the sacred process of producing an amalgamation of other people’s work—something we all do all the time; passing it through the lens of our consciousness is perhaps one of the core activities that make us human—to become available as a service.

concordDance 2y ago
> The best method for assessing performance when learning is as old as the world: assess the effort, not how well the result complies with some requirements. I am really quite confused about what you think the point of education is. In general, the world (either the physical world or the employment world) does not care about effort, it cares about results. Someone laboriously filling their kettle with a teaspoon might be putting in a ton of effort, but I'd much rather someone else make the tea who can use a tap. Why do we care about grades? Because universities and employers use them to quickly assess how useful someone is likely to be. Few people love biochemistry enough that they'd spend huge sums of money and time at university if it didn't help get them a job.
- strogonoff 2y ago
  You may be mistaking “the world” with “education” or “learning”. Producing a result is not evidence of learning progress. During learning, result is a somewhat useful metric if it roughly correlates with the level of effort, but relying only on result when determining whether to praise or reward a person during the learning stage is always a recipe for issues. A student may quickly learn to reproduce the desired result and stop progressing.
- strogonoff 2y ago
  > Someone laboriously filling their kettle with a teaspoon might be putting in a ton of effort, but I'd much rather someone else make the tea who can use a tap. By your own logic, the student who fills the kettle with the spoon has produced the expected result. Fast enough with the spoon and sky’s the limit, right? A good teacher, while praising the effort, would help them find out about the tap. Not praising the effort would give the opposite signal! You have worked hard, and through no fault of your own (no one has built-in knowledge about the tap) you were essentially told that was for nothing?! And if you have learned the tap, do you want to be done with it? Or be pushed to keep applying the same effort as with the spoon, but directed more wisely knowing that there’s a tap? Imagine what heights would you reach then! The worst teachers are in whose class 30% of the students are filling their kettle with spoons all their time, 30% simply dip them into the puddle and never get used to do the work, 30% give up because what is even the point of filling the kettle when their home has a hot water dispenser. Love your analogy, by the way.
injidup 2y ago
Little Johnny who tried really hard but still can barely write a for loop doesn't deserve a place in a comp sci course ahead of little Timmy who for some reason thinks in computer code. Timmy might be a lazy arse but he's good at what he does and for minimal effort the outcomes are amazing. Johnny unfortunately just doesn't get it. He's wanted to be a programmer ever since he saw the movie Hackers but his brain just doesn't work that way. How to evaluate this situation? Ability or effort?
- strogonoff 2y ago
  My evaluation: 1. Whoever determined that he does not “deserve” this is wrong. There may be other constraints, but no one gets to frame it as “deserves” when a child wants to learn something. 2. If a teacher is unable to teach Johnny to write a for loop, despite Johnny’s genuine utmost motivation, I would question teacher’s competence or at least fit. 3. Like any mentor, a professor in higher ed may want to choose whom to teach so that own expertise and teaching ability is realized to the fullest. Earlier in life, elementary school teacher’s luxury to do so may be limited (which is why their job is so difficult and hopefully well-compensated), and one bailing on a kid due to lack of patience or teaching competence is detestable. 4. If Johnny continues to pursue this with genuine utmost motivation, he will most likely succeed despite any incompetent teachers. If he does not succeed and yet continues to pursue this to the detriment to his life, that is something a psychologist should help him with. As for Timmy, if he learns to produce the expected result with least effort, for which he receives constant praise from the teacher, and keeps coasting this way, that does him a major disservice as far as mental mental and self-actualisation in life.
  - injidup 2y ago
    It's funny. You have created yourself a paradox. Replace comp sci with being a teacher. You have made the claim now that teachers can be incompetent but Johnny cannot be. Let's say Johnny wants to become a teacher and puts in lots of effort but just cannot teach. Now he is an incompetent teacher but at what point did he go from being judged on effort to being judged on ability? When he wanted to be a teacher and got a free pass for being a bad teacher? When he went for his first job and got a free pass for failing his exams? When his entire class learned nothing because he was unable to teach even though he put in lots of effort? Where is the transition? At some point ability is more important than effort.
tonypace 2y ago
It's fairly simple in most situations. If it doesn't involve a computer, it's handwritten in class. If it does involve a computer, it's a temporarily offline computer. We have figured out solutions to these problems already.
- strogonoff 2y ago
  You forgot “no homework that counts, or a prison- or monastery-like environment where you have no access to any of these technologies for the length of academic term”. No, humans have not ever had a similar problem before, and also some of the solutions to various problems that we have figured out in our past are no longer considered reasonable today.
  - tonypace 2y ago
    No homework that counts is essentially a double win.
    - strogonoff 2y ago
      See the part about “reasonable”. Let’s see how you single-handedly revolutionize school systems worldwide :)
- kenjackson 2y ago
  It may be that offline LLMs will be common in a few years.
  - tonypace 2y ago
    That is definitely a potential issue, but so far any text models that run on laptops are tremendously slow. Still, something to look out for.
ndriscoll 2y ago
I've found that in adulthood, I've still been judged on results, not effort, and unless we're going to drastically reduce student:teacher ratios, I don't see how you even could judge on effort. Some kids are going to learn more quickly than others, and for them, no effort will be required. At best you might assign them busywork, but that doesn't take effort just as it wouldn't take effort for an adult to do the work.
- strogonoff 2y ago
  In regular life we are all judged by others based on results, of course. When learning, however, you are best judged on effort. > Some kids are going to learn more quickly than others, and for them, no effort will be required. If no effort is required, then the bar is wrong.
  - ndriscoll 2y ago
    As long as we don't have the resources to devote 10+% of the workforce to teaching, the bar will be wrong. The bar was wrong for me during school and university, and I found teachers who gave high weights to homework or even attendance quizzes to be extremely obnoxious. On setting up expectations for adulhood, I think this is exactly backwards: > If you got good grades at school with little actual effort and the constant praise for that formed your identity, you may be in for a bad time in adulthood. Praising a child for effort without results seems way more likely to set them up for a surprisingly bad time as an adult. My personal experience has been that the "good grades/rewards without effort" thing has continued and seems pretty likely to continue through adulthood as long as you go into some kind of engineering.
    - strogonoff 2y ago
      Yes, this is a failure some or all school systems suffer from today, as I pointed out in my comment. > My personal experience has been that the "good grades/rewards without effort" thing has continued and seems pretty likely to continue through adulthood as long as you go into some kind of engineering. Based on my observation, people who are comfortable doing the work in engineering achieve completely different heights than people who got used to coasting. As applied ML spreads across industries, the difference between the competitiveness of those two categories will only become more pronounced. Furthermore, from my observation those who got used to coasting suffer from issues like perfectionism, narcissism, rejection aversion, and similar. Sooner or later in adulthood not doing the work stops achieving results deserving praise. > Praising a child for effort without results seems way more likely to set them up for a surprisingly bad time as an adult. Not “without results”. Results are critical. However, if results do not comply with whatever requirements, that is not a factor in whether to reward, unless you reward compliance. Rewarding compliance has to happen to some degree, but should not be overdone unless your goal is to foster uncreative cogs.

concordDance 2y ago

> The best method for assessing performance when learning is as old as the world: assess the effort, not how well the result complies with some requirements. I am really quite confused about what you think the point of education is. In general, the world (either the physical world or the employment world) does not care about effort, it cares about results. Someone laboriously filling their kettle with a teaspoon might be putting in a ton of effort, but I'd much rather someone else make the tea who can use a tap. Why do we care about grades? Because universities and employers use them to quickly assess how useful someone is likely to be. Few people love biochemistry enough that they'd spend huge sums of money and time at university if it didn't help get them a job.

strogonoff 2y ago
You may be mistaking “the world” with “education” or “learning”. Producing a result is not evidence of learning progress. During learning, result is a somewhat useful metric if it roughly correlates with the level of effort, but relying only on result when determining whether to praise or reward a person during the learning stage is always a recipe for issues. A student may quickly learn to reproduce the desired result and stop progressing.
strogonoff 2y ago
> Someone laboriously filling their kettle with a teaspoon might be putting in a ton of effort, but I'd much rather someone else make the tea who can use a tap. By your own logic, the student who fills the kettle with the spoon has produced the expected result. Fast enough with the spoon and sky’s the limit, right? A good teacher, while praising the effort, would help them find out about the tap. Not praising the effort would give the opposite signal! You have worked hard, and through no fault of your own (no one has built-in knowledge about the tap) you were essentially told that was for nothing?! And if you have learned the tap, do you want to be done with it? Or be pushed to keep applying the same effort as with the spoon, but directed more wisely knowing that there’s a tap? Imagine what heights would you reach then! The worst teachers are in whose class 30% of the students are filling their kettle with spoons all their time, 30% simply dip them into the puddle and never get used to do the work, 30% give up because what is even the point of filling the kettle when their home has a hot water dispenser. Love your analogy, by the way.

injidup 2y ago

Little Johnny who tried really hard but still can barely write a for loop doesn't deserve a place in a comp sci course ahead of little Timmy who for some reason thinks in computer code. Timmy might be a lazy arse but he's good at what he does and for minimal effort the outcomes are amazing. Johnny unfortunately just doesn't get it. He's wanted to be a programmer ever since he saw the movie Hackers but his brain just doesn't work that way. How to evaluate this situation? Ability or effort?

strogonoff 2y ago
My evaluation: 1. Whoever determined that he does not “deserve” this is wrong. There may be other constraints, but no one gets to frame it as “deserves” when a child wants to learn something. 2. If a teacher is unable to teach Johnny to write a for loop, despite Johnny’s genuine utmost motivation, I would question teacher’s competence or at least fit. 3. Like any mentor, a professor in higher ed may want to choose whom to teach so that own expertise and teaching ability is realized to the fullest. Earlier in life, elementary school teacher’s luxury to do so may be limited (which is why their job is so difficult and hopefully well-compensated), and one bailing on a kid due to lack of patience or teaching competence is detestable. 4. If Johnny continues to pursue this with genuine utmost motivation, he will most likely succeed despite any incompetent teachers. If he does not succeed and yet continues to pursue this to the detriment to his life, that is something a psychologist should help him with. As for Timmy, if he learns to produce the expected result with least effort, for which he receives constant praise from the teacher, and keeps coasting this way, that does him a major disservice as far as mental mental and self-actualisation in life.
- injidup 2y ago
  It's funny. You have created yourself a paradox. Replace comp sci with being a teacher. You have made the claim now that teachers can be incompetent but Johnny cannot be. Let's say Johnny wants to become a teacher and puts in lots of effort but just cannot teach. Now he is an incompetent teacher but at what point did he go from being judged on effort to being judged on ability? When he wanted to be a teacher and got a free pass for being a bad teacher? When he went for his first job and got a free pass for failing his exams? When his entire class learned nothing because he was unable to teach even though he put in lots of effort? Where is the transition? At some point ability is more important than effort.

tonypace 2y ago

It's fairly simple in most situations. If it doesn't involve a computer, it's handwritten in class. If it does involve a computer, it's a temporarily offline computer. We have figured out solutions to these problems already.

strogonoff 2y ago
You forgot “no homework that counts, or a prison- or monastery-like environment where you have no access to any of these technologies for the length of academic term”. No, humans have not ever had a similar problem before, and also some of the solutions to various problems that we have figured out in our past are no longer considered reasonable today.
- tonypace 2y ago
  No homework that counts is essentially a double win.
  - strogonoff 2y ago
    See the part about “reasonable”. Let’s see how you single-handedly revolutionize school systems worldwide :)
kenjackson 2y ago
It may be that offline LLMs will be common in a few years.
- tonypace 2y ago
  That is definitely a potential issue, but so far any text models that run on laptops are tremendously slow. Still, something to look out for.

ndriscoll 2y ago

I've found that in adulthood, I've still been judged on results, not effort, and unless we're going to drastically reduce student:teacher ratios, I don't see how you even could judge on effort. Some kids are going to learn more quickly than others, and for them, no effort will be required. At best you might assign them busywork, but that doesn't take effort just as it wouldn't take effort for an adult to do the work.

strogonoff 2y ago
In regular life we are all judged by others based on results, of course. When learning, however, you are best judged on effort. > Some kids are going to learn more quickly than others, and for them, no effort will be required. If no effort is required, then the bar is wrong.
- ndriscoll 2y ago
  As long as we don't have the resources to devote 10+% of the workforce to teaching, the bar will be wrong. The bar was wrong for me during school and university, and I found teachers who gave high weights to homework or even attendance quizzes to be extremely obnoxious. On setting up expectations for adulhood, I think this is exactly backwards: > If you got good grades at school with little actual effort and the constant praise for that formed your identity, you may be in for a bad time in adulthood. Praising a child for effort without results seems way more likely to set them up for a surprisingly bad time as an adult. My personal experience has been that the "good grades/rewards without effort" thing has continued and seems pretty likely to continue through adulthood as long as you go into some kind of engineering.
  - strogonoff 2y ago
    Yes, this is a failure some or all school systems suffer from today, as I pointed out in my comment. > My personal experience has been that the "good grades/rewards without effort" thing has continued and seems pretty likely to continue through adulthood as long as you go into some kind of engineering. Based on my observation, people who are comfortable doing the work in engineering achieve completely different heights than people who got used to coasting. As applied ML spreads across industries, the difference between the competitiveness of those two categories will only become more pronounced. Furthermore, from my observation those who got used to coasting suffer from issues like perfectionism, narcissism, rejection aversion, and similar. Sooner or later in adulthood not doing the work stops achieving results deserving praise. > Praising a child for effort without results seems way more likely to set them up for a surprisingly bad time as an adult. Not “without results”. Results are critical. However, if results do not comply with whatever requirements, that is not a factor in whether to reward, unless you reward compliance. Rewarding compliance has to happen to some degree, but should not be overdone unless your goal is to foster uncreative cogs.