4 ms·

Fascinating. I'm extremely skeptical, of course. The thing that bothers me most is that they list percentages (perhaps this is the percentage of time that the

by jibiki 18y ago

Fascinating. I'm extremely skeptical, of course. The thing that bothers me most is that they list percentages (perhaps this is the percentage of time that the rats guess correctly?) rather than $ amounts. Even if the experiment fails, explaining the fact that the later generations performed better could lead to some really interesting science.

streety 18y ago
Did the later generations perform better? As far as I could see it was only Mr. Kleinworth Morgan Jr 5 which performed noticeably better. Run through a statistical test I doubt the results observed would deviate from random variation.
- jibiki 18y ago
  I got that the 2nd generation average is 48.7% and the 3rd generation average is 50.4%. I'm too lazy to find the standard deviations.
Eliezer 18y ago
Percentages rather than dollars are worthless. There are skewed bets; you can predict "up" rightly 70% of the time and lose money on the bigger downward 30% of the movements.
- joeyo 18y ago
  Be that as it may, if the rats cannot do better than a coin toss, what use is it?
  - JabavuAdams 18y ago
    Your response indicates that you did not understand the parent post. Assume I have some system where I guess right only 25% of the time, but if I guess right I get $5. If I guess wrong I lose $1. My expected return is (0.25 * 5 - 0.75 * 1) = 0.5, which is 50 cents. So, I'm doing worse than a coin toss, but still making money. This is what the parent poster meant by a "skewed bet". The payoff for guessing "heads" or "tails" is not the same. You can't just look at the results of the coin toss (correct guess, incorrect guess). You also have to look at how much you get paid for a correct guess, and how much you lose for an incorrect guess. In most real-life situations, the payoff is not symmetric (equal win and loss amounts). This is why your "if you can't beat a coin-toss" comment is meaningless (and usually incorrect).
    - joeyo 18y ago
      I still don't get it: A. I guess correctly 25% of the time (via some method) and make money because of the skewed payoff. (0.25 * 5 - 0.75 * 1) = 0.5 B. I flip a coin and guess correctly 50% of time and make EVEN MORE money because of the skewed payoff. (0.5 * 5 - 0.5 * 1) = 2 Why should I ever go with option A? Furthermore if your method lets you guess correctly 25% of the time, why don't you simply make the opposite trade and now you are guessing correctly 75% of the time! Are we are talking about something fundamentally non-binomial? (buy, sell and do nothing or something even more complicated?)
      - JabavuAdams 18y ago
        Ah, ok. If there's a skewed payoff for B, then yes, you're correct. I read your post to mean that you were not considering the payoff, and only the probabilities. As for the rest, what does "opposite trade" mean? For instance, going short versus going long carries very different risks. It's unlikely that your payoff would simply be mirrored.
        joeyo 18y ago
        > I read your post to mean that you were not considering > the payoff, and only the probabilities. Ultimately, the payoff is just acting as a constant offset to the break-even point (assuming the payoff doesn't vary with some other parameter). The skewed payoff may mean that you only need 25% accuracy to break even or it could mean that you need 95% accuracy to break even. It doesn't matter. Either way, you can effectively ignore it and consider, for a given payoff schedule, how your prediction algorithm will perform. If we live in a universe where you can under-perform a coin toss and still make money because of how the bet is skewed, then I can do better by flipping a coin! If the bet is skewed the other way, then we will both lose money but my coin toss will lose less. Let me restate that: you can completely dissociate your prediction algorithm from your cost function > As for the rest, what does "opposite trade" mean? For > instance, going short versus going long carries very > different risks. It's unlikely that your payoff would > simply be mirrored. Here my (lack of) knowledge of the various types of financial transactions that can be made puts me at a disadvantage, but the way the "article" describes it, the rats were trained to press a green button (long, betting prices were going to go up) or a red button (short, betting prices were going to go down). There are only four outcomes here (as I understand it): * Predict Up, Moves Up * Predict Down, Moves down * Predict Up, Moves Down * Predict Down, Moves Up It's possible that you can better predict upward movements than downward movements, but lets assume for simplicity that you (or the rats) are equally bad at both. If you were able to predict at 25% accuracy, I would take what you told me (up or down) and flip it-- because you are actually performing at 75% accuracy, you just don't know it. Then I would make a trade. I don't know what the most clever trade that could be made based on that knowledge, or what the various payoffs associated with them are, but as I showed above, it doesn't matter for the analysis.

by jibiki 18y ago

streety 18y ago

Did the later generations perform better? As far as I could see it was only Mr. Kleinworth Morgan Jr 5 which performed noticeably better. Run through a statistical test I doubt the results observed would deviate from random variation.

jibiki 18y ago
I got that the 2nd generation average is 48.7% and the 3rd generation average is 50.4%. I'm too lazy to find the standard deviations.

Eliezer 18y ago

Percentages rather than dollars are worthless. There are skewed bets; you can predict "up" rightly 70% of the time and lose money on the bigger downward 30% of the movements.

joeyo 18y ago
Be that as it may, if the rats cannot do better than a coin toss, what use is it?
- JabavuAdams 18y ago
  Your response indicates that you did not understand the parent post. Assume I have some system where I guess right only 25% of the time, but if I guess right I get $5. If I guess wrong I lose $1. My expected return is (0.25 * 5 - 0.75 * 1) = 0.5, which is 50 cents. So, I'm doing worse than a coin toss, but still making money. This is what the parent poster meant by a "skewed bet". The payoff for guessing "heads" or "tails" is not the same. You can't just look at the results of the coin toss (correct guess, incorrect guess). You also have to look at how much you get paid for a correct guess, and how much you lose for an incorrect guess. In most real-life situations, the payoff is not symmetric (equal win and loss amounts). This is why your "if you can't beat a coin-toss" comment is meaningless (and usually incorrect).
  - joeyo 18y ago
    I still don't get it: A. I guess correctly 25% of the time (via some method) and make money because of the skewed payoff. (0.25 * 5 - 0.75 * 1) = 0.5 B. I flip a coin and guess correctly 50% of time and make EVEN MORE money because of the skewed payoff. (0.5 * 5 - 0.5 * 1) = 2 Why should I ever go with option A? Furthermore if your method lets you guess correctly 25% of the time, why don't you simply make the opposite trade and now you are guessing correctly 75% of the time! Are we are talking about something fundamentally non-binomial? (buy, sell and do nothing or something even more complicated?)
    - JabavuAdams 18y ago
      Ah, ok. If there's a skewed payoff for B, then yes, you're correct. I read your post to mean that you were not considering the payoff, and only the probabilities. As for the rest, what does "opposite trade" mean? For instance, going short versus going long carries very different risks. It's unlikely that your payoff would simply be mirrored.
      - joeyo 18y ago
        > I read your post to mean that you were not considering > the payoff, and only the probabilities. Ultimately, the payoff is just acting as a constant offset to the break-even point (assuming the payoff doesn't vary with some other parameter). The skewed payoff may mean that you only need 25% accuracy to break even or it could mean that you need 95% accuracy to break even. It doesn't matter. Either way, you can effectively ignore it and consider, for a given payoff schedule, how your prediction algorithm will perform. If we live in a universe where you can under-perform a coin toss and still make money because of how the bet is skewed, then I can do better by flipping a coin! If the bet is skewed the other way, then we will both lose money but my coin toss will lose less. Let me restate that: you can completely dissociate your prediction algorithm from your cost function > As for the rest, what does "opposite trade" mean? For > instance, going short versus going long carries very > different risks. It's unlikely that your payoff would > simply be mirrored. Here my (lack of) knowledge of the various types of financial transactions that can be made puts me at a disadvantage, but the way the "article" describes it, the rats were trained to press a green button (long, betting prices were going to go up) or a red button (short, betting prices were going to go down). There are only four outcomes here (as I understand it): * Predict Up, Moves Up * Predict Down, Moves down * Predict Up, Moves Down * Predict Down, Moves Up It's possible that you can better predict upward movements than downward movements, but lets assume for simplicity that you (or the rats) are equally bad at both. If you were able to predict at 25% accuracy, I would take what you told me (up or down) and flip it-- because you are actually performing at 75% accuracy, you just don't know it. Then I would make a trade. I don't know what the most clever trade that could be made based on that knowledge, or what the various payoffs associated with them are, but as I showed above, it doesn't matter for the analysis.