4 ms·

There was a Decision Analysis class at Stanford that took this idea to an extreme. For each multiple choice option, you had to indicate the percentage confidenc

by whack 10y ago

There was a Decision Analysis class at Stanford that took this idea to an extreme. For each multiple choice option, you had to indicate the percentage confidence of that option being correct. The percentages for all 4 options had to add up to 100%. The score you got for each question = 1 + ln(confidence-assigned-for-correct-answer). So if you were completely sure and gave the correct answer 100%, you would get a score of 1. If you gave it 90%, you would get a slightly lower score. If you "guessed wrong" and gave the correct answer 0% probability, ln0 is negative infinity, so you would instantly fail the entire assignment. It was a pain, but it really made you reassess how well you know stuff, rewarded people who truly knew their stuff, and burnt anyone whose confidence exceeded their actual abilities.

jacoblambda 10y ago
I have heard of this before and while it horrifies me as a student, I would love to take a class with a policy like this.
nickkoul 10y ago
In reality even in your most confident answers couldn't you put 97% on the one you believe is correct and 1% on all others ensuring that you will not receive an ln(0).
- Cyph0n 10y ago
  It actually depends on how they take the confidence value. If it's percentage, I agree with you. But if it's in numerical form - i.e., 0.01 - you'd still lose that question and a few others.
Cyph0n 10y ago
Why would anyone ever put 0% then? I would just redefine "impossible" to be an arbitrary percentage that guarantees I don't nuke the question. Quick question: is confidence taken in percent form (ln(90)) or decimal form (ln(0.9))? Edit: I am aware of the implication of having percent values, guys/gals. But if it's in decimal form, a 5% confidence level will end up losing you the current question AND three others.
- hexane360 10y ago
  If it was in percent form, a correct answer with 100% confidence would be 1 + ln(100) = 5.605 pts. Only ln(1) = 0.
- derefr 10y ago
  I assume that was part of the lesson: in Bayesian decision theory, an agent that assigns 0.0 or 1.0 confidence to something can no longer update on new evidence about that thing. Saying something is 0% likely means being so sure about that thing, that nothing—no matter how amazing—could or would ever change your mind. Basically, no human is actually ever 0.0 or 1.0 sure of anything. Assign five or ten nines of confidence, but not 0.0 or 1.0. (It's a lot clearer when you remove the log scale and notice that a 1.0 confidence-value translates to you saying you're "infinitely confident", which seems ridiculous.)
  - jumpCastle 10y ago
    But then what is my confidence that God exists? I would say zero because of the spaghetti monster argument
    - Cyph0n 10y ago
      He's right though, you can never be truly certain that some kind of God doesn't exist. But if you're referring to a specific God, then maybe.
  - hyperpape 10y ago
    I suppose this is a bad place to ask this, but given that you need to do math to update your credences and also that you can derive any proposition from Pr('2 + 2 = 4) != 1, how does this work?
    - derefr 10y ago
      Basically: probabilities and credences aren't the same thing. They're measured on the same scale, but probabilities can be used to reason about either facts or evidence, while credences are only "about" evidence (though potentially evidence about facts.) To put that another way: it is "100% true" that 2 + 2 = 4 (in ZF set theory); just like it is "100% true" that the sun will rise today, because it already did (and otherwise the Monty Hall problem would have a different answer.) But the credence in an agent's mind can and should only be informed by observations of these facts; and the "observing hardware" cannot be 1.0 trustworthy. (Your senses can be imprecise; the brain can be buggy or biased; the environment being observed can be the construction of a Cartesian "evil demon"; etc.) Given an artificial agent with perfect knowledge (e.g. one who is able to directly entangle itself with the environment it is trying to predict, rather than just one able to entangle itself with evidence about the state of an environment), some things can theoretically be held to have 0.0 or 1.0 probabilities (e.g. events that already happened)—but given an embodied agent, the "maybe my processing is flawed" argument should still make these into non-infinite credences.
hectormalot 10y ago
I had a professor that used a slightly different approach. A correct answer would yield you 1pt, but any of the 3 incorrect answers would subtract 0.5pts. Expected average result for guessing is negative, so he incentivized people to leave questions open if they were unsure.
- CaseyJParker 10y ago
  The expected average for a straight guess with absolutely zero exposure to the material would still be slightly positive. If nothing else, there's intuition and ingenuity, which place a person just a touch above a truly random guess-generating-machine.
  - hyperpape 10y ago
    With four answers, you get .25 + (.75)(-.5) = -.125 for guessing. Edit: yeah, I badly misread this.
    - cercatrova 10y ago
      That's what the parent is saying to be not applicable to the situation; humans are not simply true random guessing machines but are able to reason and eliminate choices, which would increase the expected value to be positive.
catern 10y ago
Yes, there's a class on the same subject at CMU with the same policy. Here's the document they handed out to explain the policy: http://cmu.io/~sbaugh/midterm_grading_function.pdf http://cmu.io/~sbaugh/midterm_grading_function.pdf

by whack 10y ago

jacoblambda 10y ago

I have heard of this before and while it horrifies me as a student, I would love to take a class with a policy like this.

nickkoul 10y ago

In reality even in your most confident answers couldn't you put 97% on the one you believe is correct and 1% on all others ensuring that you will not receive an ln(0).

Cyph0n 10y ago
It actually depends on how they take the confidence value. If it's percentage, I agree with you. But if it's in numerical form - i.e., 0.01 - you'd still lose that question and a few others.

Cyph0n 10y ago

Why would anyone ever put 0% then? I would just redefine "impossible" to be an arbitrary percentage that guarantees I don't nuke the question. Quick question: is confidence taken in percent form (ln(90)) or decimal form (ln(0.9))? Edit: I am aware of the implication of having percent values, guys/gals. But if it's in decimal form, a 5% confidence level will end up losing you the current question AND three others.

hexane360 10y ago
If it was in percent form, a correct answer with 100% confidence would be 1 + ln(100) = 5.605 pts. Only ln(1) = 0.
derefr 10y ago
I assume that was part of the lesson: in Bayesian decision theory, an agent that assigns 0.0 or 1.0 confidence to something can no longer update on new evidence about that thing. Saying something is 0% likely means being so sure about that thing, that nothing—no matter how amazing—could or would ever change your mind. Basically, no human is actually ever 0.0 or 1.0 sure of anything. Assign five or ten nines of confidence, but not 0.0 or 1.0. (It's a lot clearer when you remove the log scale and notice that a 1.0 confidence-value translates to you saying you're "infinitely confident", which seems ridiculous.)
- jumpCastle 10y ago
  But then what is my confidence that God exists? I would say zero because of the spaghetti monster argument
  - Cyph0n 10y ago
    He's right though, you can never be truly certain that some kind of God doesn't exist. But if you're referring to a specific God, then maybe.
- hyperpape 10y ago
  I suppose this is a bad place to ask this, but given that you need to do math to update your credences and also that you can derive any proposition from Pr('2 + 2 = 4) != 1, how does this work?
  - derefr 10y ago
    Basically: probabilities and credences aren't the same thing. They're measured on the same scale, but probabilities can be used to reason about either facts or evidence, while credences are only "about" evidence (though potentially evidence about facts.) To put that another way: it is "100% true" that 2 + 2 = 4 (in ZF set theory); just like it is "100% true" that the sun will rise today, because it already did (and otherwise the Monty Hall problem would have a different answer.) But the credence in an agent's mind can and should only be informed by observations of these facts; and the "observing hardware" cannot be 1.0 trustworthy. (Your senses can be imprecise; the brain can be buggy or biased; the environment being observed can be the construction of a Cartesian "evil demon"; etc.) Given an artificial agent with perfect knowledge (e.g. one who is able to directly entangle itself with the environment it is trying to predict, rather than just one able to entangle itself with evidence about the state of an environment), some things can theoretically be held to have 0.0 or 1.0 probabilities (e.g. events that already happened)—but given an embodied agent, the "maybe my processing is flawed" argument should still make these into non-infinite credences.

hectormalot 10y ago

I had a professor that used a slightly different approach. A correct answer would yield you 1pt, but any of the 3 incorrect answers would subtract 0.5pts. Expected average result for guessing is negative, so he incentivized people to leave questions open if they were unsure.

CaseyJParker 10y ago
The expected average for a straight guess with absolutely zero exposure to the material would still be slightly positive. If nothing else, there's intuition and ingenuity, which place a person just a touch above a truly random guess-generating-machine.
- hyperpape 10y ago
  With four answers, you get .25 + (.75)(-.5) = -.125 for guessing. Edit: yeah, I badly misread this.
  - cercatrova 10y ago
    That's what the parent is saying to be not applicable to the situation; humans are not simply true random guessing machines but are able to reason and eliminate choices, which would increase the expected value to be positive.

catern 10y ago

Yes, there's a class on the same subject at CMU with the same policy. Here's the document they handed out to explain the policy: http://cmu.io/~sbaugh/midterm_grading_function.pdf http://cmu.io/~sbaugh/midterm_grading_function.pdf