13 ms·

Shannon's Demon and how returns can be created out of thin air

63 points by tacon 4y ago

ouid 4y ago
Smoke and mirrors, the first plot is log scale, and thus demonstrates exponential growth in your portfolio. no rebalancing necessary, there no free lunch here, just a manipulator using graphs to manipulate.
- Schroedingersat 4y ago
  It's poorly communicated, but it is a real effect. You're effectively moving money from the very rare case where you win a thousand times in a row to the common case where you don't. So instead of having vanishingly rare cases where you have all the money in the universe, and common cases where you go bankrupt you have a median case where you make a little money, rare cases where you make a 2x return or similar, and common but a minority of cases where you lose most of your money. The key is that the game (1.5 vs 0.66) has a positive arithmetic expectation and zero geometric expectation, so you should be able to profit from it somehow. Do it with a double or nothing game and you always lose because the geometric expectation is negative infinity and arithmetic is zero.
pjbeam 4y ago
This reminds me of that gambling strategy where you double your bet every time to make up for losses. This seems to reduce to timing the market.
- gweinberg 4y ago
  That is called Martingale https://en.wikipedia.org/wiki/Martingale_(betting_system) https://en.wikipedia.org/wiki/Martingale_(betting_system)
deleted 4y ago
[deleted]
01100011 4y ago
Author should attempt to backtest this using historical data and see if it actually generates a return in the real world(with taxes factored in as well).
SamReidHughes 4y ago
How returns can be created out of thin air... when you have a bet with EV +8.333% of the money you put up. You can be assured that the possibilities of a portfolio balanced between a set of logarithmically random walking assets -- if one were to believe that individual stocks' behavior had such behavior -- is a well-contemplated concept in finance.
psi75 4y ago
- sharemywin 4y ago
  I kind of disagree. The article itself had interesting information. was there a little blub at the bottom, yes. but the article itself had info I hadn't seen before. I personally would have left the last paragraph off the article because I usually check out the main landing page to see what the company does or is about if the article is interesting anyway. But, as long as someone is providing an interesting point of view or new information I not that turned off by a who we are paragraph at the bottom.
  - fml_101 4y ago
    I'm not trying to be negative... The article claims the expected gain from this strategy is 2% but it's actually 0%. Plug in the numbers for all permutations of possible results if you don't believe me. Starting with a dollar and betting half we get: 1 flip H .5 gain T -.5 loss 2 flips H H = .5 + .75 = 1.25 H T = -.25 T H = -.25 T T = -.75 3 flips H H H 1.125 H H T -1.125 H T H .75/2 H T T -(.75/2) T T H .25/2 T T T -(.25/2) T H H .75/2 T H T -(.75/2) You can see the expected outcome, or average, is always 0, not .02 times the initial investment as this article claims to prove. I'm not saying rebalancing is a bad investment strategy as it reduces volatility in favor of smaller gains, but it can't generate returns on zero growth assets like fair coin flips. The writer either doesn't understand that or is misleading the audience. This is how inexperienced investors get lured into losing their life savings. It's not a joke.
    - Dylan16807 4y ago
      Those aren't the odds the article uses at all. In the article if you start with a dollar and bet 50 cents, a win gives you +25 cents and a loss gives you -16.7 cents. If you don't rebalance, the median outcome is that your 50 cents stays 50 cents forever. If you do rebalance, the median outcome is steady growth.
gweinberg 4y ago
You don;t necessarily have to be able to spot the trick to know there is one. If you know that the expectation value of any combination of bets is just the sum of the expectation values of the original bets, you'll know there cannot be a way to combine zero-expectation bets to get a positive expectation. Because no matter how you sum up zeroes, you'll end up with zero.
- saurik 4y ago
  I recommend watching this lecture, as the effect here isn't magic or fake: it is just a tradeoff of how you manage your portfolio to convert volatility into return and there is a ton of well-studied math relating to how it works. https://youtu.be/8TJQhQ2GZ0Y https://youtu.be/8TJQhQ2GZ0Y And like, yes: there is a moment in the lecture where he explicitly sets up a scenario where the expectation value of your bets sums to zero and yet the rebalancing makes money. It isn't a free lunch, of course: if the two assets both go down and stay down then you lose money, and you don't gain as much money as you'd want on the way up; but if they are both sputtering around you definitely make money. If you were to somehow be able to rebalance infinitely fast it also doesn't make money; and yet, it only makes money in relationship to the volume you are able to rebalance, so there is another tradeoff. At the end of the day you can thereby model it as making money by providing the service of "providing liquidity" to the market, and what you are effectively doing is taking a spread on that as a market maker.
  - s1artibartfast 4y ago
    Is it just a fancy Martingale then?
    - Dylan16807 4y ago
      A Martingale that you can do with finite money sounds pretty nice on the surface.
      - s1artibartfast 4y ago
        You can Martingale with finite money too.
cochne 4y ago
Quick summary: > To illustrate this concept, you could consider a simple coin-flip game where you make a 50% return on your money if the coin lands on heads, or lose 33.3% of your money if the coin lands on tails. Let's say you bet $100. Your expected outcome is 100 * (1/2 * 3/2 + 1/2 * 2/3) = 108.33. Greater than 100. That's all there is to it, the "magic" comes from the made up security which can generate such amazing returns.
- curtisf 4y ago
  There's actually a little something here, but the blog loses it by being way too loose with "expectation". The interesting thing, which is confused by the blog, is that "expectation" is not that same thing as "expectation of log". The "expectation of log" is a simple and effective way to adjust for risk, so trading off between these two can be useful. In the simple betting case, you're multiplying your money by {e^k, e^-k}. In this case, k≈0.4, so you end up with Y = P * {0.66, 1.50} This has expectation of `1.08 * P`, and it has an expectation-of-log of `log P`. In the rebalancing case, you're multiplying your money by (1 + {e^k, e^-k})/2. You end up with Z = P * {0.83, 1.25} This has an expectation of `1.04 * P`, and it has an expectation-of-log of `log P + 0.0204`. So you actually have a smaller expectation, but a larger expectation-of-log. This could be helpful, depending on your goals, because it is a simple way to describe how "risky" the strategy is. For example, after 5 rounds of this, the first strategy has a median value of +8.5%, while the second strategy has a median of +13%. On the other hand, on average the first strategy has grown +49% while the second has grown only +23%.
motohagiography 4y ago
Is this not what hedgefunds were designed specifically to execute for clients? Presumably using this strategy as their own internal beta or benchmark, and then finding uncorrelated volatile assets and trades to execute the same thing to generate alpha for themselves, and charging ~2/20 fee struture on it. Literally value machines. Hence their demand for exotic products with uncorrelated volatility characteristics for risk pairing. I superficially get it now. I can see how someone could get the bug for this.
- notacoward 4y ago
  > Literally value machines. Only for a definition of "value" that seems pretty highly questionable on a bunch of different levels.
alpineidyll3 4y ago
Pairs trades are amongst the most common elementary quant strategies, which is why all equities are so damn correlated and which destroys the apparent arb described in the article.
- imtringued 4y ago
  In fact, the article simply explains the origin of arbitrage from a mathematical perspective.
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
fml_101 4y ago
This is why gambling is stupid.. Do everyone a favor and invest in yourself - real skills to give back to the world. “can actually make it possible to generate positive returns at the portfolio level, even when all of the portfolio’s individual holdings are expected to produce zero, or even negative, returns over time.” -False The illusion comes from the fact that, for small n you can flip heads and tails the same number of times and still have over half the money you started with - whereas with the invest all strategy you would have lost everything. For a fair coin, as n approaches infinity you still win nothing either way though. That graph is completely misleading. We can represent the amount won as the recursive sequence X_(n+1) = X_n + (X_n)/2 def recursive_sequence(n): """ Money won after winning n times in a row""" result = 1 # money we start with rebalance_ratio = .5 for i in range(0,n): result += result*rebalance_ratio return result for i in range(1,10): temp_profit = recursive_sequence(i) net = temp_profit / 2**i # assuming you loose as many times as you won after percent = abs((temp_profit-1))*100 print(f"savings after {i} heads:", recursive_sequence(i), f"\npercent profit: {percent}","\n....money after loosing n times after: ", net, "\n") You can see what this gives for a rebalance ratio of 0.5. Anything more or less just increases or decreases your volatility. “How returns can be created out of thin air” is literally a scam - not true. That’s a tulip bubble. Sure you can get rich off one if you fool enough people. You need to flip a coin 2^(n+1) – 2 times to get heads n times in a row. Same for tails… Your “rebalancing” is only increasing the number of attempts it takes to not win anything ie. the time you waste flipping coins instead of spending time with people. I feel embarrassed for spending as much time as I did on this. At least I learned something… savings after 1 heads: 1.5 percent profit: 50.0 ....money after loosing n times after: 0.75 savings after 2 heads: 2.25 percent profit: 125.0 ....money after loosing n times after: 0.5625 savings after 3 heads: 3.375 percent profit: 237.5 ....money after loosing n times after: 0.421875 savings after 4 heads: 5.0625 percent profit: 406.25 ....money after loosing n times after: 0.31640625 savings after 5 heads: 7.59375 percent profit: 659.375 ....money after loosing n times after: 0.2373046875 savings after 6 heads: 11.390625 percent profit: 1039.0625 ....money after loosing n times after: 0.177978515625 savings after 7 heads: 17.0859375 percent profit: 1608.59375 ....money after loosing n times after: 0.13348388671875
ComplexSystems 4y ago
There is a much easier way to grasp what is happening here. Often in finance, securities are assumed to follow something like a geometric Brownian pattern. This is the idea behind the Black-Scholes model, for instance. This means that you view your price ticker as a random Gaussian walk (e.g. Brownian motion) when your ticker is placed on a logarithmic scale. One way to think of this is that, given some (sufficiently large) unit of time, you have an equal chance of your security either doubling or halving in value. Or being multiplied/divided by 1.5, or 1.1, or any other figure you like, given whatever unit of time you like. It's really easy to see how this basic pattern has a positive expected value. Suppose you put $1 into a fair coin toss game, and if you win it doubles and if you lose it halves. You stand to win $1 or lose only $0.5, thus the expected value is $1.25. Now suppose, to keep it simple, you have a ridiculously volatile security which doubles or halves in value every day. So the ideal trading strategy is to choose some betting amount, say $1000, and just put it in at the start of the day. Then the next day, just reset your position size to $1000, either taking your $1000 in profit or kicking in another $500 to cover losses. You will quickly become a trillionaire.
- amluto 4y ago
  > So the ideal trading strategy is to choose some betting amount, say $1000, and just put it in at the start of the day. Then the next day, just reset your position size to $1000, either taking your $1000 in profit or kicking in another $500 to cover losses. You will quickly become a trillionaire. No, you will quickly become a thousandaire, because your strategy is very very far from optimal. You will gain an expected $250 per day, your earnings will not compound at all, and you will wonder why you are not ultra-rich at the end of the year. Of course, this is better than getting all your money every day, which, in your example, will net you nothing on expectation: the expected change of the log of your assets every day is exactly zero. If you actually use the Kelly criterion, you will indeed become very wealthy very quickly.
amluto 4y ago
As far as I can tell, the term “Shannon’s demon” is fairly recently made up. I learned this as the Kelly criterion. Here’s the original paper: https://www.princeton.edu/~wbialek/rome/refs/kelly_56.pdf https://www.princeton.edu/~wbialek/rome/refs/kelly_56.pdf To “Shannon’s demon’s” credit, the first word of that paper is Shannon, but that’s about it. Kelly and Shannon were different people, and the paper makes no mention of demons. This also has nothing to do with making returns out of thin air. It’s about how to avoid losing your returns by betting too much at once, and how to optimally choose the size of a bet.

63 points by tacon 4y ago

ouid 4y ago

Smoke and mirrors, the first plot is log scale, and thus demonstrates exponential growth in your portfolio. no rebalancing necessary, there no free lunch here, just a manipulator using graphs to manipulate.

Schroedingersat 4y ago
It's poorly communicated, but it is a real effect. You're effectively moving money from the very rare case where you win a thousand times in a row to the common case where you don't. So instead of having vanishingly rare cases where you have all the money in the universe, and common cases where you go bankrupt you have a median case where you make a little money, rare cases where you make a 2x return or similar, and common but a minority of cases where you lose most of your money. The key is that the game (1.5 vs 0.66) has a positive arithmetic expectation and zero geometric expectation, so you should be able to profit from it somehow. Do it with a double or nothing game and you always lose because the geometric expectation is negative infinity and arithmetic is zero.

pjbeam 4y ago

This reminds me of that gambling strategy where you double your bet every time to make up for losses. This seems to reduce to timing the market.

gweinberg 4y ago
That is called Martingale https://en.wikipedia.org/wiki/Martingale_(betting_system) https://en.wikipedia.org/wiki/Martingale_(betting_system)

deleted 4y ago

[deleted]

01100011 4y ago

Author should attempt to backtest this using historical data and see if it actually generates a return in the real world(with taxes factored in as well).

SamReidHughes 4y ago

How returns can be created out of thin air... when you have a bet with EV +8.333% of the money you put up. You can be assured that the possibilities of a portfolio balanced between a set of logarithmically random walking assets -- if one were to believe that individual stocks' behavior had such behavior -- is a well-contemplated concept in finance.

psi75 4y ago

sharemywin 4y ago
I kind of disagree. The article itself had interesting information. was there a little blub at the bottom, yes. but the article itself had info I hadn't seen before. I personally would have left the last paragraph off the article because I usually check out the main landing page to see what the company does or is about if the article is interesting anyway. But, as long as someone is providing an interesting point of view or new information I not that turned off by a who we are paragraph at the bottom.
- fml_101 4y ago
  I'm not trying to be negative... The article claims the expected gain from this strategy is 2% but it's actually 0%. Plug in the numbers for all permutations of possible results if you don't believe me. Starting with a dollar and betting half we get: 1 flip H .5 gain T -.5 loss 2 flips H H = .5 + .75 = 1.25 H T = -.25 T H = -.25 T T = -.75 3 flips H H H 1.125 H H T -1.125 H T H .75/2 H T T -(.75/2) T T H .25/2 T T T -(.25/2) T H H .75/2 T H T -(.75/2) You can see the expected outcome, or average, is always 0, not .02 times the initial investment as this article claims to prove. I'm not saying rebalancing is a bad investment strategy as it reduces volatility in favor of smaller gains, but it can't generate returns on zero growth assets like fair coin flips. The writer either doesn't understand that or is misleading the audience. This is how inexperienced investors get lured into losing their life savings. It's not a joke.
  - Dylan16807 4y ago
    Those aren't the odds the article uses at all. In the article if you start with a dollar and bet 50 cents, a win gives you +25 cents and a loss gives you -16.7 cents. If you don't rebalance, the median outcome is that your 50 cents stays 50 cents forever. If you do rebalance, the median outcome is steady growth.

gweinberg 4y ago

You don;t necessarily have to be able to spot the trick to know there is one. If you know that the expectation value of any combination of bets is just the sum of the expectation values of the original bets, you'll know there cannot be a way to combine zero-expectation bets to get a positive expectation. Because no matter how you sum up zeroes, you'll end up with zero.

saurik 4y ago
I recommend watching this lecture, as the effect here isn't magic or fake: it is just a tradeoff of how you manage your portfolio to convert volatility into return and there is a ton of well-studied math relating to how it works. https://youtu.be/8TJQhQ2GZ0Y https://youtu.be/8TJQhQ2GZ0Y And like, yes: there is a moment in the lecture where he explicitly sets up a scenario where the expectation value of your bets sums to zero and yet the rebalancing makes money. It isn't a free lunch, of course: if the two assets both go down and stay down then you lose money, and you don't gain as much money as you'd want on the way up; but if they are both sputtering around you definitely make money. If you were to somehow be able to rebalance infinitely fast it also doesn't make money; and yet, it only makes money in relationship to the volume you are able to rebalance, so there is another tradeoff. At the end of the day you can thereby model it as making money by providing the service of "providing liquidity" to the market, and what you are effectively doing is taking a spread on that as a market maker.
- s1artibartfast 4y ago
  Is it just a fancy Martingale then?
  - Dylan16807 4y ago
    A Martingale that you can do with finite money sounds pretty nice on the surface.
    - s1artibartfast 4y ago
      You can Martingale with finite money too.

cochne 4y ago

Quick summary: > To illustrate this concept, you could consider a simple coin-flip game where you make a 50% return on your money if the coin lands on heads, or lose 33.3% of your money if the coin lands on tails. Let's say you bet $100. Your expected outcome is 100 * (1/2 * 3/2 + 1/2 * 2/3) = 108.33. Greater than 100. That's all there is to it, the "magic" comes from the made up security which can generate such amazing returns.

curtisf 4y ago
There's actually a little something here, but the blog loses it by being way too loose with "expectation". The interesting thing, which is confused by the blog, is that "expectation" is not that same thing as "expectation of log". The "expectation of log" is a simple and effective way to adjust for risk, so trading off between these two can be useful. In the simple betting case, you're multiplying your money by {e^k, e^-k}. In this case, k≈0.4, so you end up with Y = P * {0.66, 1.50} This has expectation of `1.08 * P`, and it has an expectation-of-log of `log P`. In the rebalancing case, you're multiplying your money by (1 + {e^k, e^-k})/2. You end up with Z = P * {0.83, 1.25} This has an expectation of `1.04 * P`, and it has an expectation-of-log of `log P + 0.0204`. So you actually have a smaller expectation, but a larger expectation-of-log. This could be helpful, depending on your goals, because it is a simple way to describe how "risky" the strategy is. For example, after 5 rounds of this, the first strategy has a median value of +8.5%, while the second strategy has a median of +13%. On the other hand, on average the first strategy has grown +49% while the second has grown only +23%.

motohagiography 4y ago

Is this not what hedgefunds were designed specifically to execute for clients? Presumably using this strategy as their own internal beta or benchmark, and then finding uncorrelated volatile assets and trades to execute the same thing to generate alpha for themselves, and charging ~2/20 fee struture on it. Literally value machines. Hence their demand for exotic products with uncorrelated volatility characteristics for risk pairing. I superficially get it now. I can see how someone could get the bug for this.

notacoward 4y ago
> Literally value machines. Only for a definition of "value" that seems pretty highly questionable on a bunch of different levels.

alpineidyll3 4y ago

Pairs trades are amongst the most common elementary quant strategies, which is why all equities are so damn correlated and which destroys the apparent arb described in the article.

imtringued 4y ago
In fact, the article simply explains the origin of arbitrage from a mathematical perspective.

deleted 4y ago

[deleted]

deleted 4y ago

[deleted]

fml_101 4y ago

This is why gambling is stupid.. Do everyone a favor and invest in yourself - real skills to give back to the world. “can actually make it possible to generate positive returns at the portfolio level, even when all of the portfolio’s individual holdings are expected to produce zero, or even negative, returns over time.” -False The illusion comes from the fact that, for small n you can flip heads and tails the same number of times and still have over half the money you started with - whereas with the invest all strategy you would have lost everything. For a fair coin, as n approaches infinity you still win nothing either way though. That graph is completely misleading. We can represent the amount won as the recursive sequence X_(n+1) = X_n + (X_n)/2 def recursive_sequence(n): """ Money won after winning n times in a row""" result = 1 # money we start with rebalance_ratio = .5 for i in range(0,n): result += result*rebalance_ratio return result for i in range(1,10): temp_profit = recursive_sequence(i) net = temp_profit / 2**i # assuming you loose as many times as you won after percent = abs((temp_profit-1))*100 print(f"savings after {i} heads:", recursive_sequence(i), f"\npercent profit: {percent}","\n....money after loosing n times after: ", net, "\n") You can see what this gives for a rebalance ratio of 0.5. Anything more or less just increases or decreases your volatility. “How returns can be created out of thin air” is literally a scam - not true. That’s a tulip bubble. Sure you can get rich off one if you fool enough people. You need to flip a coin 2^(n+1) – 2 times to get heads n times in a row. Same for tails… Your “rebalancing” is only increasing the number of attempts it takes to not win anything ie. the time you waste flipping coins instead of spending time with people. I feel embarrassed for spending as much time as I did on this. At least I learned something… savings after 1 heads: 1.5 percent profit: 50.0 ....money after loosing n times after: 0.75 savings after 2 heads: 2.25 percent profit: 125.0 ....money after loosing n times after: 0.5625 savings after 3 heads: 3.375 percent profit: 237.5 ....money after loosing n times after: 0.421875 savings after 4 heads: 5.0625 percent profit: 406.25 ....money after loosing n times after: 0.31640625 savings after 5 heads: 7.59375 percent profit: 659.375 ....money after loosing n times after: 0.2373046875 savings after 6 heads: 11.390625 percent profit: 1039.0625 ....money after loosing n times after: 0.177978515625 savings after 7 heads: 17.0859375 percent profit: 1608.59375 ....money after loosing n times after: 0.13348388671875

ComplexSystems 4y ago

There is a much easier way to grasp what is happening here. Often in finance, securities are assumed to follow something like a geometric Brownian pattern. This is the idea behind the Black-Scholes model, for instance. This means that you view your price ticker as a random Gaussian walk (e.g. Brownian motion) when your ticker is placed on a logarithmic scale. One way to think of this is that, given some (sufficiently large) unit of time, you have an equal chance of your security either doubling or halving in value. Or being multiplied/divided by 1.5, or 1.1, or any other figure you like, given whatever unit of time you like. It's really easy to see how this basic pattern has a positive expected value. Suppose you put $1 into a fair coin toss game, and if you win it doubles and if you lose it halves. You stand to win $1 or lose only $0.5, thus the expected value is $1.25. Now suppose, to keep it simple, you have a ridiculously volatile security which doubles or halves in value every day. So the ideal trading strategy is to choose some betting amount, say $1000, and just put it in at the start of the day. Then the next day, just reset your position size to $1000, either taking your $1000 in profit or kicking in another $500 to cover losses. You will quickly become a trillionaire.

amluto 4y ago
> So the ideal trading strategy is to choose some betting amount, say $1000, and just put it in at the start of the day. Then the next day, just reset your position size to $1000, either taking your $1000 in profit or kicking in another $500 to cover losses. You will quickly become a trillionaire. No, you will quickly become a thousandaire, because your strategy is very very far from optimal. You will gain an expected $250 per day, your earnings will not compound at all, and you will wonder why you are not ultra-rich at the end of the year. Of course, this is better than getting all your money every day, which, in your example, will net you nothing on expectation: the expected change of the log of your assets every day is exactly zero. If you actually use the Kelly criterion, you will indeed become very wealthy very quickly.

amluto 4y ago

As far as I can tell, the term “Shannon’s demon” is fairly recently made up. I learned this as the Kelly criterion. Here’s the original paper: https://www.princeton.edu/~wbialek/rome/refs/kelly_56.pdf https://www.princeton.edu/~wbialek/rome/refs/kelly_56.pdf To “Shannon’s demon’s” credit, the first word of that paper is Shannon, but that’s about it. Kelly and Shannon were different people, and the paper makes no mention of demons. This also has nothing to do with making returns out of thin air. It’s about how to avoid losing your returns by betting too much at once, and how to optimally choose the size of a bet.