2 ms·

I understand the math. The outcome of a single-shot instruction has an x% chance of being right that has nothing to do with the boolean outcome of either right

I understand the math. The outcome of a single-shot instruction has an x% chance of being right that has nothing to do with the boolean outcome of either right or wrong, and is almost never (and only coincidentally) 50%. It does not make sense to assume a 50% chance of success for an instruction like: Tell me what day of the week it is, tell me what town in the world my mom was born, tell me who/what Homer claimed to be the progeny of Dionysus to be, tell me which stock will perform best in the S&P tomorrow, tell my what time I will arrive in Tokyo, tell me how many stars there are in the Milky Way, etc.

lossolo 1y ago
You keep treating the count of wrong labels as if it automatically fixes the probability of being wrong. That is only true when you have already assumed a uniform distribution. Nothing in probability theory forces that assumption. Let Ω = {y₁, y₂, …, yₙ} (sample space = the labels) y ∈ Ω (the single correct label) P : Ω → [0,1] with ∑ᵢ P(yᵢ)=1 (a probability measure) Define two events Correct = {y} Wrong = Ω \ {y} Then P(Correct) = P({y}) = P(y) Because P is arbitrary apart from normalisation, we are free to set: P(y) = 0.50 P(any other yᵢ) = 0.50 / (n-1) That instantly gives P(Correct) = 0.50, P(Wrong) = 0.50. The outcome space collapses to a Bernoulli(½) coin-flip no matter whether n = 2 or n = 10⁹. Going back to your marble example: 99 black marbles (wrong) 1 red marble (right) Uniform draw => 1% success. But "uniform" is a weight choice. Make the red marble 99x heavier (or magnetic, or add 98 dummy red slips you ignore when grading): P(red) = 99 / (99+99) = 0.50 P(black) = 1 / (99+99) ≈ 0.00505 Same 100 physical marbles, now 50 % success. The count of wrong ways (99) never changed, only the weights did. It matters for ML classifiers because every multiclass classifier ultimately gets scored on accuracy, i.e. Pr(output = ground-truth) That accuracy is exactly P(Correct) above. The model’s internal distribution over labels (learned or engineered) determines that number. Uniform guessing over 100 labels gives 1% accuracy. A better model might concentrate 50% mass on the right label and reach 50% accuracy, which is literally a coin-flip in outcome space even though 99 wrong labels remain. As to your strawman, I never said every real world question lands at 50%. I said: if a system places 0.5 probability mass on the correct label, then its success odds are 50%, FULL STOP. Whether that distribution is realistic for "What day of the week is it?" or "Which stock will lead the S&P tomorrow?" is an empirical question but it has nothing to do with the mere fact that there are many wrong answers. Probability theory says that success is whatever probability mass you assign to the single correct label, the label count is irrelevant once the distribution is non-uniform. The short math proof above settles that and ends our discussion.

lossolo 1y ago

You keep treating the count of wrong labels as if it automatically fixes the probability of being wrong. That is only true when you have already assumed a uniform distribution. Nothing in probability theory forces that assumption. Let Ω = {y₁, y₂, …, yₙ} (sample space = the labels) y ∈ Ω (the single correct label) P : Ω → [0,1] with ∑ᵢ P(yᵢ)=1 (a probability measure) Define two events Correct = {y} Wrong = Ω \ {y} Then P(Correct) = P({y}) = P(y) Because P is arbitrary apart from normalisation, we are free to set: P(y) = 0.50 P(any other yᵢ) = 0.50 / (n-1) That instantly gives P(Correct) = 0.50, P(Wrong) = 0.50. The outcome space collapses to a Bernoulli(½) coin-flip no matter whether n = 2 or n = 10⁹. Going back to your marble example: 99 black marbles (wrong) 1 red marble (right) Uniform draw => 1% success. But "uniform" is a weight choice. Make the red marble 99x heavier (or magnetic, or add 98 dummy red slips you ignore when grading): P(red) = 99 / (99+99) = 0.50 P(black) = 1 / (99+99) ≈ 0.00505 Same 100 physical marbles, now 50 % success. The count of wrong ways (99) never changed, only the weights did. It matters for ML classifiers because every multiclass classifier ultimately gets scored on accuracy, i.e. Pr(output = ground-truth) That accuracy is exactly P(Correct) above. The model’s internal distribution over labels (learned or engineered) determines that number. Uniform guessing over 100 labels gives 1% accuracy. A better model might concentrate 50% mass on the right label and reach 50% accuracy, which is literally a coin-flip in outcome space even though 99 wrong labels remain. As to your strawman, I never said every real world question lands at 50%. I said: if a system places 0.5 probability mass on the correct label, then its success odds are 50%, FULL STOP. Whether that distribution is realistic for "What day of the week is it?" or "Which stock will lead the S&P tomorrow?" is an empirical question but it has nothing to do with the mere fact that there are many wrong answers. Probability theory says that success is whatever probability mass you assign to the single correct label, the label count is irrelevant once the distribution is non-uniform. The short math proof above settles that and ends our discussion.