4 ms·

Why is the 20 multiplied by the million? Why not 0.95 multiplied by million? Using the same argument I would accept infinite odds that my username is quickthro

Why is the 20 multiplied by the million? Why not 0.95 multiplied by million? Using the same argument I would accept infinite odds that my username is quickthrower2 so there is infinite information?

Strilanc 3y ago
It's important to understand that when they said "bits" they didn't mean information in the Shannon entropy sense, but rather in the log-odds evidence sense. Gaining a Shannon entropy bit means learning the answer to a yes-no question that had 1:1 odds. Gaining a log-odds evidence bit means doubling your best-guess odds on a question you are uncertain about, from X:Y to (2X):Y. One Shannon bit is worth arbitrarily many evidence bits, because a Shannon bit takes you from 1:1 odds to UNBOUNDEDLYHUGE:1 odds. So... yeah, actually, reading your username is worth infinite bits of log-odds evidence on what your username is! (Ignoring practical issues like the small chance of computer malfunctions, of course.) And to answer your initial question: the 20 just came from the assertion they'd bet 20:1. That was arbitrary.
- neilkk 3y ago
  This isn't how the mathematics of odds works, as the GP correctly pointed out. An event which is 20:1 on is not 20 times more likely than certainty. Going from 1:4 to 1:2 means that the event has become twice as likely. But going from 2:1 to 4:1 does not: it means that the complementary event has become half as likely. Based on this, we can't do math with odds treating them identically to ratios. If you do the math correctly, the two types of information measure are basically the same thing.
  - Strilanc 3y ago
    A log-odds of b bits means an odds of 2^b : 1 which means a probability of p = 2^b / (2^b + 1). In the original comment, the evidence update was stated as going from 20:1 to 1:1000000 and it was claimed this was approximately 24 bits of evidence. The update is from 2^4.3:1 to 2^-19.9:1. Subtracting the exponents you get 4.3 - -19.9 = 24.2 which is approximately 24 as claimed. The "20" in 20:1 is correctly accounted for by the ~4 additional bits of evidence on top of updating from 1:1000000 to 1:1. Clearly evidence bits behave very differently from entropy bits. Acquiring a single entropy bit is an update from 1:1 to 0:1 which is 2^0:1 to 2^-infinity:1. It's worth an unbounded number of evidence bits. It's important not to mix these two things up.
    - neilkk 3y ago
      Yes, you are doing math with odds as though they are fractions or ratios, which is deeply incorrect. 20:1 is not the reciprocal of 1:20 but the complement. Odds ratios and similar calculations do not work like this. You can do this type of calculation using X:1 odds or 1:X odds but not both in the same calculation. Or perhaps you can provide a reference to a justification of this type of calculation?
      - Strilanc 3y ago
        Search "decibels" in https://www.yudkowsky.net/rational/bayes https://www.yudkowsky.net/rational/bayes for the explanation. I think you're just wrong about needing everything to be in the form X:1 or 1:X. When I compute the ratio of 1000000:1 divided by 1:20 it gives 1000000:(1/20) then scaling both sides by the same factor gives 20000000:1.
        neilkk 3y ago
        Your reference definitely doesn't show a calculation of the type you are trying to do. Likelihood ratios are not the same as odds ratios; they do not have the problem I described. I would be very surprised if you can find any reference at all to the number you describe as 'evidence bits', or anything equivalent, made by anyone who can show an understanding of basic probability, statistics, or information theory. I understand how you get 20,000,000 as the answer to the calculation you carry out. My point is that that number is not meaningful in any way.
        Strilanc 3y ago
        When you apply a statistical test, the various outcomes cause Bayesian updates that correspond to adding or subtracting fixed bits of evidence. When you repeat the test (and the repetitions are independent), the amount of bits of evidence you add or subtract remain the same. In other words, focusing on bits of evidence shows Bayesian updates behave like a biased random walk under repetition of a test and allow you to compute the properties of that walk. For example, suppose you are trying to estimate how much rounding errors in a pseudo random number generator betray that it is not a true exact representation of the random process. One way to quantify this is to compute the expected bits of evidence revealed per call to the RNG.

Why is the 20 multiplied by the million? Why not 0.95 multiplied by million? Using the same argument I would accept infinite odds that my username is quickthrower2 so there is infinite information?

Strilanc 3y ago

It's important to understand that when they said "bits" they didn't mean information in the Shannon entropy sense, but rather in the log-odds evidence sense. Gaining a Shannon entropy bit means learning the answer to a yes-no question that had 1:1 odds. Gaining a log-odds evidence bit means doubling your best-guess odds on a question you are uncertain about, from X:Y to (2X):Y. One Shannon bit is worth arbitrarily many evidence bits, because a Shannon bit takes you from 1:1 odds to UNBOUNDEDLYHUGE:1 odds. So... yeah, actually, reading your username is worth infinite bits of log-odds evidence on what your username is! (Ignoring practical issues like the small chance of computer malfunctions, of course.) And to answer your initial question: the 20 just came from the assertion they'd bet 20:1. That was arbitrary.

neilkk 3y ago
This isn't how the mathematics of odds works, as the GP correctly pointed out. An event which is 20:1 on is not 20 times more likely than certainty. Going from 1:4 to 1:2 means that the event has become twice as likely. But going from 2:1 to 4:1 does not: it means that the complementary event has become half as likely. Based on this, we can't do math with odds treating them identically to ratios. If you do the math correctly, the two types of information measure are basically the same thing.
- Strilanc 3y ago
  A log-odds of b bits means an odds of 2^b : 1 which means a probability of p = 2^b / (2^b + 1). In the original comment, the evidence update was stated as going from 20:1 to 1:1000000 and it was claimed this was approximately 24 bits of evidence. The update is from 2^4.3:1 to 2^-19.9:1. Subtracting the exponents you get 4.3 - -19.9 = 24.2 which is approximately 24 as claimed. The "20" in 20:1 is correctly accounted for by the ~4 additional bits of evidence on top of updating from 1:1000000 to 1:1. Clearly evidence bits behave very differently from entropy bits. Acquiring a single entropy bit is an update from 1:1 to 0:1 which is 2^0:1 to 2^-infinity:1. It's worth an unbounded number of evidence bits. It's important not to mix these two things up.
  - neilkk 3y ago
    Yes, you are doing math with odds as though they are fractions or ratios, which is deeply incorrect. 20:1 is not the reciprocal of 1:20 but the complement. Odds ratios and similar calculations do not work like this. You can do this type of calculation using X:1 odds or 1:X odds but not both in the same calculation. Or perhaps you can provide a reference to a justification of this type of calculation?
    - Strilanc 3y ago
      Search "decibels" in https://www.yudkowsky.net/rational/bayes https://www.yudkowsky.net/rational/bayes for the explanation. I think you're just wrong about needing everything to be in the form X:1 or 1:X. When I compute the ratio of 1000000:1 divided by 1:20 it gives 1000000:(1/20) then scaling both sides by the same factor gives 20000000:1.
      - neilkk 3y ago
        Your reference definitely doesn't show a calculation of the type you are trying to do. Likelihood ratios are not the same as odds ratios; they do not have the problem I described. I would be very surprised if you can find any reference at all to the number you describe as 'evidence bits', or anything equivalent, made by anyone who can show an understanding of basic probability, statistics, or information theory. I understand how you get 20,000,000 as the answer to the calculation you carry out. My point is that that number is not meaningful in any way.
        Strilanc 3y ago
        When you apply a statistical test, the various outcomes cause Bayesian updates that correspond to adding or subtracting fixed bits of evidence. When you repeat the test (and the repetitions are independent), the amount of bits of evidence you add or subtract remain the same. In other words, focusing on bits of evidence shows Bayesian updates behave like a biased random walk under repetition of a test and allow you to compute the properties of that walk. For example, suppose you are trying to estimate how much rounding errors in a pseudo random number generator betray that it is not a true exact representation of the random process. One way to quantify this is to compute the expected bits of evidence revealed per call to the RNG.