18 ms·

Is Probability Real?

220 points by EbTech 6y ago

EbTech 6y ago
Since this topic isn't so well-known, I wrote the case arguing that frequentist interpretations don't work, but algorithmic information theory (Kolmogorov complexity) does. I want to make this accessible and persuasive, so thoughts, questions, and arguments would be appreciated!
- opheliate 6y ago
  Such an interesting post, thank you for sharing! I'm in my second year of a maths degree currently, and we obviously studied frequentist probability/stats in the first year, but I'm not taking any probability modules this year. I found the tone & accessibility was just right for me :)
- enriquto 6y ago
  A simple sentence that I've found useful for pedagogy: "the probability of that coin toss being 50% does not talk about the coin; it talks about you, and about your partial knowledge of the universe." You can add: "The coin toss itself is deterministic and the result can be computed if you know the initial position and speed." They will inevitably bother you about the physical impossibility to measure the starting position and speed exactly, and then you say "ok, forget about the coin. You have 5 white and 5 black balls inside this opaque cylinder. What's the probability that the top ball is white? This does not talk about the balls (the color of the top one is already determined) but about your partial knowledge of them". (EDIT: formatting)
  - alisonkisk 6y ago
    How does the ball situation help? It has the same problems of physical impossibility of measuring however the balls were ordered. (Modelling someone's brain?) I guess the argument works on someone with an unscientific model of the human brain, but that's one step forward and two steps back.
    - enriquto 6y ago
      Somebody just put the balls there carefully, and did not tell you in what order, just how many of each color.
      - Frost1x 6y ago
        That still boils down to a lack of prior information which I don't think removes the argument for "I don't have enough information." Probably have to use actual quantum phenomena that behave probabilistically by definition if you want a currently irrefutable physical example. I'm personally not convinced even this is fundamentally probabilistic and we currently have to rely on probability theory as a crutch for complex behaviors we just quite don't understand yet or don't have the time and resources to compute.
        posterboy 6y ago
        You don't need quantum physics to formulate a philosophical standpoint that happens to agree with the Kopenhagen Interpretation. I'm not sure if it helps, but I suppose a compromise here would be the assumption that you don't really know the starting configuration of yourself, why you draw probabilistic inferences naturally, that the sun will go up tomorrow like every day. If that has a biologic explanation, then the top comment was not just to the illusive argument of platonic ideals.
        enriquto 6y ago
        > "I don't have enough information." My point exactly. Probability theory is a precise mathematical formalization of the concept of "not enough information".
  - st1x7 6y ago
    > the probability of that coin toss being 50% does not talk about the coin; it talks about you, and about your partial knowledge of the universe. But it does talk about the coin - a weighted coin would have a different probability. Same in the example with the white/black balls - if they weren't 5 white and 5 black but 6 white and 4 black, the probability you would assign to the top one would be different. Again, the probability is a way to describe the balls themselves, not just our knowledge. I get the general idea of representing probability as uncertainty and partial knowledge but your statements strike me as just straight up incorrect.
    - wearsshoes 6y ago
      This is still a partial knowledge situation - you have the information that there are a certain proportion of colored balls in the chamber, but not the information about their order. The probability includes the information we do know and allows inferences about information we don’t know.
      - st1x7 6y ago
        Sure, I just can't agree with the parent statement that probability has nothing to do with the object it describes, which is demonstrably false.
        StavrosK 6y ago
        I don't think it's demonstrably false: If you don't know that the coin is weighted, the probability is 50%. Probabilities are predictions and estimates, not fundamentally about the thing itself, but about what we know about the thing.
        deleted 6y ago
        [deleted]
        mannykannot 6y ago
        The principle of indifference? I know that it is a commonplace assumption, but feels to me as though one is assuming one has more information than is justified. Coming back to the article's "economist's wager", is it rational to bet with even odds on something you know nothing about? If the assumption is interpreted as a testable hypothesis about outcomes, why would complete ignorance imply any particular result? On the other hand, if it is interpreted strictly as a statement about one's knowledge, why present it exactly as if one had sufficient knowledge of the situation to know that the probability is 0.5? Maybe the author will have an answer in part 2.
  - Sharlin 6y ago
    This is basically the Bayesian interpretation of probability.
    - enriquto 6y ago
      Of course. But if you pronounce a fancy word like "bayesian" there's a large amount of minds that shut irremediably.
      - v64 6y ago
        That's also why we call it QBism [1] instead of quantum bayesianism [1] https://en.wikipedia.org/wiki/Quantum_Bayesianism https://en.wikipedia.org/wiki/Quantum_Bayesianism
    - lottin 6y ago
      Saying that "the probability of a coin toss of 50% talks about you" is not an interpretation of probability. Saying that we are "50% sure" is also not an interpretation of probability. It's a nonsensical statement. It's like saying we are "50% angry". It doesn't really mean anything.
      - alisonkisk 6y ago
        I don't understand your claims that these statements are are meaningless. They are commonly uttered and understood.
        lottin 6y ago
        I can understand expressions such as "pretty sure" or "completely sure". I do not understand the expression "to be X% sure". If someone says they're "37% sure" tomorrow will rain, what does that mean exactly?
        dinosaurdynasty 6y ago
        37% of the time that someone says they are 37% sure of a statement X the statement X is true (assuming they're calibrated correctly/etc).
  - nimbleal 6y ago
    It may be deterministic, but are you sure it would be computable? One does not necessarily imply the other.
  - kps 6y ago
    Forget the coin and the balls — does the nucleus decay? You're not missing any knowledge; there isn't any.
    - jbay808 6y ago
      This is definitely the most interesting example, but it's not obvious that a situation where the relevant information is fundamentally inaccessible is a situation where you aren't missing any information. It's your best bet for a scenario where you can be sure that nobody else has more information than you do, though.
  - mannykannot 6y ago
    Wigner's friend might have something to say about this... I don't have a specific argument to make here, only the feeling that if it were all just a matter of what a given observer knows, no-one would be talking about there being a QM measurement problem.
    - codethief 6y ago
      This is a very good point! In fact, some people do argue that there is no measurement problem in the Copenhagen formulation of quantum mechanics to begin with – at least if you take it seriously and strictly go by the rule that the laws laid down by Bohr et al. only concern you as the observer and your knowledge about the system, and not the system itself. Following this train of thought, there is nothing "real" about the wavefunction and it is just a tool to come up with predictions. The same goes for the collapse of the wave function (which just describes a change in your ability to predict future measurements, and not a change of the object) and the term "measurement" (which we might as well replace with "enlightenment", i.e. the moment in which we obtain knowledge about the system). In that sense, the only difference between classical and quantum mechanics is that our knowledge (viewed as a mathematical quantity) behaves differently in both theories: In classical physics, when we conduct multiple measurements of a given system in a row, our knowledge about that system will increase – to the point that, once we have measured all system properties to sufficient accuracy, we'll able to predict what any future measurement of any of those properties will yield (again, with some predictable uncertainty). So the knowledge of all our measurements has added up, it is an additive quantity. In QM, this is fundamentally different: We can only know anything about the object the very moment we look at it. The rules of quantum mechanics (again, in the very strict interpretation laid out above) dictate that the second we conduct a measurement, we can forget about any knowledge obtained through previous measurements of other (conjugate) observables: Future measurements of those observables are inherently unpredictable. In that sense, our knowledge about quantum-mechanical objects never "adds up" to anything. (To see that this is really the the distinguishing feature between classical and quantum mechanics, recall that the existence of conjugate observables really is the only thing setting apart the quantum from the classical world: Without conjugate observables it would be impossible to distinguish, say, 100 electrons in a superposition of spin up and down from an ensemble of 100 electrons of which 50 are in a spin up state and the other 50 are in a spin down state.) Of course, this whole interpretation is very unsatisfactory to lots of people (myself included) for a whole bunch of reasons. I assume that, to a large degree, this is due to the fact that laws of nature that put human observers in their very center seem rather undesirable. (At least since the time we switched from a geocentric to a heliocentric view of the world.) But my impression is that there's another reason: Our intuition from classical mechanics & statistics has taught us that objects exist independently of us as observers and behave in a deterministic fashion, at least provided we as observers know enough about them. (Meaning that the more we know about the coin's initial position and velocity, the more likely we are to predict the outcome of the coin toss. If we don't know anything about the coin, though, the outcome is as unpredictable as measuring spin up/down in quantum mechanics.) Unfortunately, this whole line of argument is circular: The reason we believe that the existence of physical objects is independent of us, is precisely because knowledge in classical mechanics is an additive quantity and we can get to the point where we know "enough" to come up with deterministic predictions. That is, we never have to discard knowledge when running new measurements and so our knowledge takes on a independent "role" – which we call reality.
  - MereInterest 6y ago
    As a caveat, while this intuition works for classical mechanics, it does not work for quantum mechanics. All observations are consistent with wave function collapse being fundamentally random. Any hidden variables would need to be transmitted many times faster than the speed of light (~10000x, last time I checked the experiments), and are therefore inconsistent with our understanding of special relativity.
    - smallnamespace 6y ago
      Note that pilot wave theory, an (out of vogue) interpretation of quantum mechanics, also recasts the apparent randomness in quantum mechanics as due to our ignorance of the exact state of the pilot wave. Even Einstein struggled with quantum mechanics, famously saying "[God] does not play dice with the universe".
  - JamisonM 6y ago
    There seems to be a lot of quibbling about the simple sentence here but I find it clarifying. Discussing coin tosses is a thought experiment with a very practical physical analog so spelling out clearly what the thought experiment's actual subject is has valuable properties so you don't get lost in the weeds of the physical execution of flipping coins.
  - deleted 6y ago
    [deleted]
- alisonkisk 6y ago
  Why is frequentism bad because it only gives certainty for infinite samples, but complexity is good despite being non-computable? It's two sides of the same coin -- computable uncertainty va non-computable certainty.
  - EbTech 6y ago
    I don't think frequentism is "bad"; just insufficient as a gold standard interpretation of probabilistic claims. I liked an analogy from the reference by Rathmanner & Hutter: the most "correct" chess-playing program involves a complete search along the tree of possible games. In practice, we try to approximate this ideal. In the case of Kolmogorov complexity, a reasonable takeaway might be to use the shortest program that we're able to find, even if it's not the shortest overall.
- TheOtherHobbes 6y ago
  OK - so apply Kolmogorov complexity to election polling. How does that work out? I think you're confusing various possible maps with the territory in a less than useful way. Given that frequentist interpretations are approximations - and understood as such - and Kolmogorov complexity isn't computable at all, what problem have you solved here?
  - EbTech 6y ago
    Hm I admit it's hard to talk convincingly about election prediction, since we don't have practical algorithms to do this; a lot of it comes down to human judgment. The philosophical point (which might be approximated algorithmically someday, or by intelligent minds today) is that your election probabilities should come out of an overall highly compressed model of the world. In theory, a Bayesian who uses the prior 2^-K(x) over all strings x should, with sufficient life experience, come up with good estimates, in a certain sense. I'll have to think about this example more carefully when fulfilling my promise of writing about how this theory relates to everyday decision-making. Thanks for pointing out a potential weakness :)
- qsort 6y ago
  The article is excellent, congratulations. A couple observations/questions. 1) You didn't comment on the bayesian viewpoint that probability reflects a subjective idea about the state of the world. One might argue, for example, that probability isn't measurable, and that therefore, strictly speaking, a statement about the objective probability of an event isn't meaningful. Experimental evaluation would have to be done on an entire model instead. Do you have any objections to that point of view? 2) I don't find the case about Kolmogorov complexity to be actually convincing, at least not as per the requirements the rest of the article sets. "3141592..." could pass as either "random digits" or "first digits of pi". The fact that it's highly unlikely a true RNG would have generated exactly those, we are back to a frequentist argument there. It's likely I'm missing something, could you elaborate more or give me a pointer?
  - SmooL 6y ago
    Isn't that what the occam's razor argument was for? Sure, both an RNG and the "40 digits of pie" program can produce that output, but the "40 digits of pie" program is shorter, therefore having less kolmogorov complexity
    - qsort 6y ago
      Yeah but is it? They are both programs with logarithmic Kolmogorov complexity, and Kolmogorov complexity is only defined up to constant factors unless you commit to a computational model. If you do commit to one, which is the shortest is just a function of what are the specifics of the model you chose, which isn't really interesting, it's literally code golf at that point.
      - FartyMcFarter 6y ago
        > They are both programs with logarithmic Kolmogorov complexity In the case of true random numbers, how is that so? Very few random sequences can be generated by a logarithmic-sized program, since most strings are not significantly compressible [1]. A simple counting argument shows that: there are 2^n strings of n bits, but only 2^(lg n) = n logarithmic-sized strings, a much smaller number! [1] http://theory.stanford.edu/~trevisan/cs154-12/kolcomplexity-rev.pdf http://theory.stanford.edu/~trevisan/cs154-12/kolcomplexity-...
- JamisonM 6y ago
  Some thoughts on the composition: * "I wrote the case arguing that frequentist interpretations don't work, but algorithmic information theory does": if that's what you are up to here then I think it would be for readers if you stated that up front in some way. And hit me with some kind of summary at the end that makes the concise version of your argument at the end, it's a long article. * Shorter might be better: There's a lot of stuff in here that I think you can pare out in the probability discussion that maybe isn't adding that much to your argument. I think there is a lot to be gained by assuming a generous reader. * Betting might be a distraction to your point: This might be confusing the imperfect knowledge of participants in a market with the imperfect knowledge of all the physical forces involved in a physical phenomenon and how that related to the seeming "randomness" of a coin flip for your reader. (The liquidity and stuff.. this is just not related to your point.) * Don't undermine your point with unrelated assumptions: "I imagine they wouldn’t consider their world unlikely at all: they would just add a new law to their description of physics: all dice, as if by divine intervention, are deemed to exhibit this strange behaviour" this lead me to think that you were just sort of shooing away the whole last X decades of high vs. low energy physics, we collectively certainly don't think that we have the rules correct precisely because of this complication, we find the idea that we need 2 sets of rules improbable and believe that there must be a way to explain everything with a single set of rules. So your mythical dice society probably would consider their dice exception a very unlikely world.. they would be confident they have the world wrong! A dubious assertion (or at least one that would need a whole lot of explanation) can be an off-ramp for a subset of readers.
  - EbTech 6y ago
    Thanks for the detailed critique! I'll take some time to think about how to better make the points that I wanted to convey with those sections.
- richard_todd 6y ago
  I would have liked to know what the Kolmogorov approach has to say about the examples used to deflate frequentism ("which of my friends will start a business?"). I don't see from the article how the "smallest-program" approach could say anything useful about those, either. Maybe that wasn't the point--but after poking holes in the frequentist view, it uses unrelated examples like digits of pi to illustrate the Kolmogorov idea, so I'm left unable to directly compare the kinds of statements the two approaches can make. Even going back to dice or coins would have helped me compare them. Like, I know frequentists can show how the variance in coin-toss outcomes decreases as the sample size increases. What can the smallest-program approach say about that? Or was the point that those variant outcomes aren't "real" enough to talk about? Does that mean there is a connection to constructivism in mathematics here? It seems either approach benefits from more data, and there must be a concept related to a "confidence interval" where, as 100, then 200, then 300 digits of pi roll in, your pi-program stays the same size while other programs have to keep growing to accommodate the new data. Like, the ratio of the smallest program to the naive encoding ought to say something about how potentially predictive the small program is. Thanks for the interesting article. It definitely made me think about the issues, and now I'm curious to know more about the topic.
  - EbTech 6y ago
    Thanks! I hope to better address your concerns in Part 2.
  - PeterisP 6y ago
    One aspect of Kolmogorov approach is that implicitly models things like biased probabilities through compressibility. The shortest representation of rolls of fair dice or coins is their exact results, but if there's "less randomness" in some way (biased coin/die, sum of two dice which means non-uniform probabilities, combination of some predictable pattern with random noise) then there are more compact representations of that information, and all of that gets captured by the Kolmogorov approach without any explicit handling of the various possibilities.
- analog31 6y ago
  My only thought is that discovering a workable definition of "scientific method" is a reach. Philosophers spent a century searching for such a thing, in vain. On the other hand, providing something that just works would be beneficial enough, even if falling short of the philosophical holy grail, so it's worth pursuing. I'm a physicist, and physicists have always wondered why math works so well in physics. There's this famous essay by Eugene Wigner: https://www.dartmouth.edu/~matc/MathDrama/reading/Wigner.html https://www.dartmouth.edu/~matc/MathDrama/reading/Wigner.htm...
- topsycatt 6y ago
  There were a few points that I, as someone unfamiliar with many of the ideas presented, got hung up on. First, the paragraph that begins with "At first blush, the requirement to use..." Seems to be a non sequitur. I don't fully understand how the previous section creates a requirement to use deterministic programs, so I could use more explanation on how that requirement is established. Second, a very simple concrete example of what one of these programs would look like would be immensely helpful. After re-reading the article a bit I have a mental image of a program that contains a long, compressed string and a decompression algorithm that somehow models the system you're interested in. I can imagine how you might get a useful interpretation of probability from the decompression system, but there are enough open questions there that I'm not sure I have the correct interpretation. Hope that helps!
  - EbTech 6y ago
    Thanks. I should clarify that the computer is deterministic, so as to avoid building randomness into the definition of randomness! I skimmed over an example too quickly, but your intuition is about right. For that sequence, two possible programs are: - Compute and print the first 40 digits of pi. - Decompress the following string according to a Shannon code with probabilities (1/36,1/18,1/12,[etc]): [insert code]
- carapace 6y ago
  This is awesome! Good work! Are you going to touch on Chaitin's Omega?
  - EbTech 6y ago
    Thanks! :) I wasn't planning to go there! While I enjoy the idea, for now I'm trying to focus on what's needed to make sense of the problem of induction. Is there a nice connection that I missed?
- danabo 6y ago
  It sounds like you are arguing that i.i.d. frequentism doesn't work. I view AIT as generalizing frequentism to non-i.i.d. timeseries. This is formalized as Martin-Lof tests for randomness, and Solomonoff induction.
REALiSTiC 6y ago
Solid write up. If anyone's interesting in knowing more about how probability _came to be_ 'regarded' as real, i highly recommend Ian Hacking's "The Taming of Chance". Thought provoking in so many ways.
MauranKilom 6y ago
> The scientific method only works because the rules of the universe happen to be simple, while the set of observations it offers is vast. Kolmogorov complexity captures this defining characteristic of our reality. I've never seen this spelled out so beautifully!
- willis936 6y ago
  Not everything that is true is simple or catchy. Biasing towards it is a trap. PBS SpaceTime has a good discussion on this topic. https://youtu.be/xFKgIOX8IRE https://youtu.be/xFKgIOX8IRE
  - jhardy54 6y ago
    Neat video, but it specifically disagrees with your point. He advocates for applying scientific rigor between leaps oc intuition. There's nothing wrong with GP appreciating beauty.
    - willis936 6y ago
      My point isn’t contradicted by the video. My point is not "beauty should not be appreciated". My point is that we should not dismiss models of nature based on aesthetics. Nature has no preference on what we see as beautiful. The video makes the point that beauty should be treated as a guiding principle, not as a hard and fast rule to sniff out truth.
- karmakaze 6y ago
  The patterns in our observations appear simple, the rules are deeper: e.g. newtonian motion vs relativistic.
- deleted 6y ago
  [deleted]
- ordu 6y ago
  It is a funny statement, I like it. But due to a different reason: all we know about reality is our theories. How we could state, that rules of the universe are simple? As I see it, we could state, that our theories are full of simple rules. But the universe have no theories nor rules outside of human's mind. It is completely our inventions, our dreams, our hopes that the universe have some rules. We could state that our simple rules works, but what does it mean "to work"? For example, a spider sees reality not like us, it feels vibrations of it's web, runs to a source of vibrations and start to bite, to wrap intruding object with web. It would do it to a tuning fork, if you pressed it to spider's web. His simple rules of reality works though. Despite the fact that sometimes spider bites steel of a tuning fork without any benefits for the spider. How could we know that our theories not just extended version of spider's? With the same issues, like they make us to do something absolutely pointless. How could we evaluate this fact? To ask our theories? But our theories already predicted that this pointless thing we do would be a good thing. We might ask our theories again and we'd get the same answer. This statement seems as a tautology for me. Our rules are simple, because they are simple. Our theories work because they tell us, that they work.
shrubble 6y ago
Probably...
rcthompson 6y ago
I'm glad to see a write-up of this. I've been searching recently for a way to reasonably define probability without having to invoke either hypothetical infinitely repeated experiments or placing bets (since the latter is really just implicitly invoking the former). I'm looking forward to part 2!
- peterwoerner 6y ago
  Check out the first chapter of Jayne's probability theory. It's the clearest take down of the frequentist interpretations that you have and introduction to the Bayesian interpretation. http://www.med.mcgill.ca/epidemiology/hanley/bios601/GaussianModel/JaynesProbabilityTheory.pdf http://www.med.mcgill.ca/epidemiology/hanley/bios601/Gaussia...
  - rcthompson 6y ago
    Thanks, I had a quick skim through it and it seems really helpful. If I understand correctly, the claim is that the laws of probability as they are known comprise the only possible interpretation that satisfies the 3 desiderata set forth in chapter 1.
    - peterwoerner 6y ago
      I don't remember, the thing which really helped me click at least the bayesian interpretation was if we have A --> B and B, then A is more plausible and probability reasoning is a way to quantify how more plausible A is now that we know B is true. Chapter 5 is also a doozy in terms of explaining some of the things going on currently.
dandanua 6y ago
It's complex! (according to quantum mechanics)
xwdv 6y ago
Ultimately, probability must be real, because if you zoom in to a low enough level, the movement of particles is basically random, and this randomness is enough to poison any deterministic explanations for which side a coin flip will land. You could calculate which side a coin will land if you knew all the variables and thus arrive at a higher probability that it will land on a certain side, but it’s still a probability, because the particles that make up the coin could all move in such a way at any point that causes the coin to fall in a way you didn’t expect, even if that probability is very low. A small probability unlikely to happen is still a probability nonetheless. Try computing how a dice made up of a few particles will land. Good luck...
- mellosouls 6y ago
  I haven't read the article either but skimmed it briefly for a later read; I think it isn't disputing the phenomena that probability is ultimately used to describe (as you seem to claim), but how they are to be interpreted.
- walleeee 6y ago
  Philip McShane expressed this nicely: "...a thing is defined by... systematizations of coincidental aggregates of the properties of lower things". Probability theory "allows for the emergence of the systematic from the non-systematic".
thread_id 6y ago
It probably depends on the p-value.
FinanceAnon 6y ago
"All models are wrong, but some are useful"
deleted 6y ago
[deleted]
smitty1e 6y ago
> “Given competing programs whose outputs match our observations, always prefer the shortest.” I need to know context. Output is necessary, but not sufficient. Is this a stand-alone chat server for some niche voices, or, say, Twitter?
loup-vaillant 6y ago
Tow things. First, probability is real: it's a construct in one's mind, and minds are just as real as dice or coins. https://www.lesswrong.com/posts/f6ZLxEWaankRZ2Crv/probability-is-in-the-mind https://www.lesswrong.com/posts/f6ZLxEWaankRZ2Crv/probabilit... Second, (and the author may be leading up to this), there's Solomonoff's theory of inductive inference, which he has proven complete: when we apply Occam's razor (where the prior probability of each possible theory drops exponentially with its size), the amount of error a perfect Bayesian makes as they observe event and bet on the next one, ad infinitum, is finite. Roughly proportional to the complexity of the simplest theory that correctly predict the whole sequence of events. It's one of the most convincing proofs that Bayesian reasoning works. https://en.wikipedia.org/wiki/Solomonoff's_theory_of_inductive_inference https://en.wikipedia.org/wiki/Solomonoff's_theory_of_inducti... There's just a little snag. Perfect Bayesian reasoning is impossible to compute, so us mortals have to resort to approximations. Just as perfect certainty isn't possible, perfect reasoning is not attainable. Oh well.
- smallnamespace 6y ago
  > the amount of error a perfect Bayesian makes as they observe event and bet on the next one, ad infinitum, is finite. There's a little assumption that you're leaving out, namely that the Kolmogorov complexity of the data generating process is finite. From Wikipedia: > expected cumulative errors made by the predictions based on Solomonoff's induction are upper-bounded by the Kolmogorov complexity of the (stochastic) data generating process Whether the universe (or our observations of the universe) have finite complexity is very much an unresolved philosophical question.
  - yters 6y ago
    If quantum is really random, then our universe has infinite Kolmogorov complexity.
    - fny 6y ago
      Is Newtonian mechanics real? Well, its not strictly real, but it's apparently real enough for engineering cars, planes, and rockets. So even if quantum is really random, I'd bet an unbiased coin will still land on heads with 50% probability every time.
cjkarr 6y ago
Maybe?
forrestthewoods 6y ago
The best part of this is that dice are known to not be fair. There’s a whole niche for people who want to buy fair dice. I can’t find it but I once saw a post that stacked ~20 d20 dice. The difference in height based on what number you picked to stack was shocking. The dice were incredibly non-uniform.
HelloNurse 6y ago
Mostly well argued, but there's some very loose language, such as calling "inconsistencies" mere practical relevance issues (such as nonideal markets, nonprobabilistic decisions, ignorance and the ensuing arbitrage, changes in probability values). A bad theory can be logically inconsistent, but it is not the case of probability theory and its competing interpretations.
jmount 6y ago
One way to deal with probability is: defer interpretation until after one has axiomatized calculation by measure theory. I have a quick video on the topic here: https://youtu.be/DnTTAd1TDyQ https://youtu.be/DnTTAd1TDyQ .
- alisonkisk 6y ago
  Measure theory is a powerful tool, but how does that answer the question of what probability is? Akin to physics, studying the wave equation doesn't tell you what the wave equation is in the real world.
karmakaze 6y ago
I don't think "first forty digits of pi" should be admissible. As it depends on an external definition that's not computed by the representation. I could come up with a mathematical definition for any prefix of digits, give it a name and say the 'first x digits of C'.
- alisonkisk 6y ago
  Complexity is defined by size over an entire class of objects, not a single one. Pi is reused many many times at constant cost, wile your C becomes C2, C3, C4,...
- skybrian 6y ago
  That's just because it's a loose description in English. It would actually be a short computer program that calculates pi, and instead of the first 40 digits you could ask for, say, a megabyte of data, or take the limit as the amount of data grows. Transmitting a program to compute pi would be shorter than the data needed by any compression algorithm that isn't somehow based on knowing the trick. The same trick could be used for any mathematically interesting number. The point is that incompressible random sequences exist that are not like that. You can't do better than transmitting the sequence itself.
  - karmakaze 6y ago
    Yes the point was merely 'referring to an external' isn't a good example of a minimal size description.
emerged 6y ago
Since the lowest levels of our understanding of the universe are probabilistic, it seems reasonable to assume probability is the most real thing we currently know. It might be that will change some day, but at the moment everything we take to be non-probabilistic is just a simplification of underlying probabilities.
leephillips 6y ago
Nice article, on the whole, and usefully provocative. But I have misgivings about making these close connections between information theory and scientific theory-making. As everyone knows, information theory leaves out any notion of semantics, as it should. But the important thing about our theories of the world is that they have meaning to us. The scientist searches for something that makes sense of the world, not an algorithm for computing a series of numbers. The theories that we search for may not have the smallest Kolmogorov complexity; the criteria that they satisfy go a lot deeper.
- texasbigdata 6y ago
  That’s such a great phrase: usefully provocative. Thanks for that.
TaurenHunter 6y ago
I like to think that probability is the ratio unknown/known, information available divided by all possibilities. You know that a coin has 2 faces (known=2) and that if you toss it, 1 face will be up (unknown which=1). The ratio is more about something in the mind than intrinsic to the objects. That helped me understand why in the Monty Hall problem, when you switch doors, the probability of getting the prize increases.
alexpetralia 6y ago
> "Somehow, we must narrow down our hypotheses. Maybe you think that’s easy: only a few hypotheses describe plausible dice behavior; the rest are patently absurd! But now you’re relying on intuitive judgment, not a rigorous methodology." Maybe, as philosopher Robert Pirsig theorized in "Zen and the Art of Motorcycle Maintenance", you must rely on Quality!
jbeam 6y ago
Very interesting article that cuts to the core of many of the issues with pricing insurance products. While the phrase is hardly limited to the actuarial world, "all models are wrong, but some are useful" is definitely an extension of this. The fluid nature of probability is the center of the insurance universe. Probability is always a moving target in the insurance world. Indeed, if it weren't, there wouldn't be much of a need for actuaries. Much of actuarial training revolves around the idea of credibility -- how credible is your sample set, what alterations should you make to old data to make it relevant to today, and what data should you add to it as a complement in order to relieve the model of the biases inherent in your sample size. This is inherently Bayesian in it's approach. Where it truly gets interesting is that insurance companies are very cognizant of tail risk -- the 1-in-100, 1-in-250, 1-in-500 events that can cause insurer insolvency if not properly accounted for. You can survive a miscalculated loss trend within reasonable bounds, but if you haven't thought about the potential Cat 5 hurricane that hits Miami-Dade then you are going to have some very unhappy investors. When it comes to these types of events, you mostly need to be in the right ballpark. The order of magnitude matters more than the exact number -- albeit the exact number matters quite a bit for regulatory reasons. This type of calculation for property lines has largely been outsourced to the stochastic models developed by companies such as AIR and RMS. A sudden change in their models, which I think is likely after this record breaking hurricane season, can inflict capital pressure on the industry almost instantly.[1] There are some actuarial papers from around 50 years ago that discuss information entropy as another way to approach the issue of constructing probability models, but they never really caught on. It seems that is likely due to the lack of widespread computing power. I'm hoping these ideas can gain some steam now that we can construct some of these distributions from Python and R. [1] There is a fantastic article by Michael Lewis that describes this issue at great detail: https://www.nytimes.com/2007/08/26/magazine/26neworleans-t.html https://www.nytimes.com/2007/08/26/magazine/26neworleans-t.h...
- sxzxs 6y ago
  For me, it comes down to what kinds of these risky models are most interesting. Some can be interesting because of potential profit or minimizing loss (financial or actuarial) and others are inherently (theoretical physics). To add to your tail risk point - I wonder how many people foresaw the Venezuelan oil crisis way back when, or even less likely, the Saudi Arabian oil complex attack in 2019. And of course, the current situation we're in with CoVID that an entire university of forward thoughtful looking people didn't call until it was a week away. As an aside, do insurance companies significantly alter their policies when such a cat-5 hurricane is imminent? What preparations would they make in the face of that sort of event? Are you talking about chaos theory in the last paragraph? I'll read that article you linked in a bit and see what more I have to say, from skimming through it looks as though my question from the previous paragraph may be answered.
nieve 6y ago
Isn't declaring "the raison d’être of probability theory is to explain the decision-making of individuals facing uncertainty" basically a claim that the sole purpose of probability theory is an economist's approach? It's entirely possible that much more work in probability is done without a bet or payoff in sight than is done for the sake of decision-making models. It seems like basing epistemological arguments about all probability theory on a framing friendly to very specific groups of non-mathematicians.
- glial 6y ago
  What is a probability? Nearly always, a probability isn’t a statement about the world, it’s a statement about your knowledge and the information you have. Even for frequentist statisticians, equating probabilities with proportions of outcomes is an admission that you only have partial knowledge about the outcomes. Of course it also has mathematical structure and properties that may be interesting to people for their own sake. And there may be interesting things to say about quantum physics using probability, but I think a historian of mathematics would not claim that quantum physics was the driving force for the development of probability theory. Anyway, the “raison d’être” != every conceivable use.
- alisonkisk 6y ago
  Can you give an alternate definition for probability than "decision making under uncertainty?" I've never heard one. Sure there are mathematical real-analytic "theories of probability" but that's abstract analysis of measure, until it's applied to answer how... probable something is
kolbe 6y ago
I work with probably as it relates to financial markets, and my gripe with probably is at an even lower level. Even with dice, the fact of the matter is that you will roll dice with almost entirely Newtonian forces acting on it, and it will settle on a number. This is not unknown to physics. It’s just unknown to us. And because we don’t have the data and computational power to know with any certainty where the dice will land, we call it random. Everything I do is centered around creating probability distributions of where a stock will be in in the future. I don’t do this because it is fundamentally unknowable, but because I cannot access all of the data necessary to know. So, I’ve come to regard probably measure as an interesting and useful tool, but one that has no connection to the reality of existence.
- EbTech 6y ago
  Fair enough! I'd like to point out that the Kolmogorov complexity approach can make sense of subjective probability too. Since you lack precise enough information to predict the dice roll, the most compressed way to write down your observations will involve a Shannon-style code with your subjective probabilities. If you have enough information but not enough computational power, the resource-bounded variants of Kolmogorov complexity may be more applicable.
  - jtsuken 6y ago
    Can you please define Shannon-style code? (Obviously, you don't have the shannon code, the compression method, in mind). Also, isn't Kolmogorov complexity uncomputable and you run into multiple "who shaves the barber" issues, when trying to determine it?
    - EbTech 6y ago
      The compression code can be specified first. If you have a lot of data, the specification will be negligible in length, compared to the code itself. Together, the specification and the Shannon code give an upper bound on the Kolmogorov compmlexity. If this is the shortest known program, we may consider the Shannon code probabilities as our "best explanation" of the data. You can also get posterior probabilities using a universal prior such as 2^-K(x), but of course, this can only be approximated in the limit of infinite runtime.
kempbellt 6y ago
The article probably has more value than my comment.
nolite 6y ago
its pretty complex
abdulhaq 6y ago
Surely you have to define reality before approaching this question?
- Koshkin 6y ago
  I wonder if it is even possible. In math, science, and philosophy some (basic) terms are necessarily left undefined and their meaning is either left open to interpretation or assumed to be evident based on common experience or convention.
ikeboy 6y ago
Probability is not real. Probability is subjective, it depends on what you know, and everyone has a different set of things they know. I flip a coin and look at it, then ask two other people for their probabilities. One of them knows the coin is biased towards head such that it's twice as likely to land heads than tails on any given flip. The other knows nothing about the coin. The first person guesses a 66% chance of heads. The second guesses a 50% chance of heads. I, having seen the coin, say it's a 0% chance of heads. None of these probabilities are wrong. They're all correct given the set of knowledge that person had. Probability is subjective.
- xvedejas 6y ago
  Your example suggests to me the opposite: Probability is a real and objective way to describe the information you have. "Garbage in, garbage out" still applies when you have no information.
  - ikeboy 6y ago
    The distinction between objective and subjective collapses under your usage.
    - kgwgk 6y ago
      https://www.youtube.com/watch?v=X5cQcmAtjJ0 https://www.youtube.com/watch?v=X5cQcmAtjJ0
    - xvedejas 6y ago
      No, the distinction is clear. If your assigned probability is something that someone else can reproduce using the same steps given the same information, then it is objective (yet contextual). Your examples each have a clear reasoning behind the assigned probabilities, they're not just opinion-based assertions.
      - ikeboy 6y ago
        Bob has a very simple algorithm to output probabilities. He just answers 50/50 for any yes or no question. This is reproducible. Is this objective?
grandrew 6y ago
I have a question about how Kolmogorov complexity is related to "Human complexity" or complexity of understanding? E.g. a program may be very complex in Kolmogorov terms, like describing 1000 random numbers - easy to understand: you have a database of numbers, and a simple procedure that would scan through it. You can also imagine some real-world microservices-based program with a good architecture and a lot of code that handles all the exception cases of incoming data, all easily understandable. And now imagine an optimizing compiler for prolog programs. It may have much less code but the algorithm will be so complex that it might be impossible to fully understand its behaviour. E.g. fast-downward is a great example of such a program. So I'm wondering what does Kolmogorov complexity actually tell? Or does it tell anything useful in "real"-world?
- jdc 6y ago
  What is easy or useful for one person to understand is often much less so for another.
- renewiltord 6y ago
  Well, that's altering the 'language' that you're measuring the KC against. For instance, in the language that CS majors use the phrase "Kolmogorov Complexity" is sufficient to encode the concept of KC itself. And in the context of that language plus the concept of KC and adding in this entire thread, the letters KC themselves encode Kolmogorov Complexity in entirety. So the 'real world' version isn't so easy to tell in absolute terms like that because the languages can differ. But if you were thinking of it like a compressor/decompressor pair, then moving things into the language is like moving things into the compressor/decompressor. Naturally then you conclude that the "human complexity" depends greatly on the humans since any pair of humans creates a new universal description language that we can see as some base language plus the jargon that they are both familiar with.
  - qayxc 6y ago
    > in the language that CS majors use the phrase "Kolmogorov Complexity" is sufficient to encode the concept of KC itself Wait, isn't that just conflating "language" with "knowledge"/"information"? The underlying assumption here is that the CS major has an association of a concept encoded by the letters "Kolmogorov Complexity". This is not universal, though, i.e. there's no computation that could derive the meaning behind these letters from the encoding alone. It's like claiming "620" is sufficient to encode Mozart's "Die Zauberflöte" ("The Magic Flute"), because in the language of a musician, the Köchel catalogue number along with the context would enable them to decode the full meaning. But in reality you would still have to look up the number and the score somewhere so it's not really an encoding but more of a pointer or index. I'd see any technical term that way, in that the term itself is not an encoding, but a key/index/identifier of a concept, not a full definition of the concept itself.
roenxi 6y ago
Nice article, there is an even easier interpretation though: The mathematics of probability defines a bunch of objects that do not exist, anywhere. Random variables, expectations, probabilities, maybe a few others. Then there are situations in the real world that look a lot like those objects from certain perspectives. It is a bit like Escher's Ascending and Descending - sometimes people make things in the real world that look like the infinite staircase when viewed from the right spot. Similarly, sometimes we find things that look probabilistic (dice rolls, coin flips) when viewed in ignorance of the sum totality of the entire universe. That is why there are a bunch of statistical tests that determine if outcomes are distinguishable from a hypothetical process generating that outcome. tl; dr; I like the article, but it seems to answer it's own headline with "yes" and it is much easier to answer it with "no". There is, philosophically speaking, nothing that we can guarantee looks random from all perspectives.
- analog31 6y ago
  Perhaps the most useful connection is that statistics lets us study what kind of math we can do, when we know something about a set, but not everything about it. And our experience with the world is like that too.
- burrows 6y ago
  > There is, philosophically speaking, nothing that we can guarantee looks random from all perspectives. Does this imply that the universe is deterministic? I believe it is false to claim “Humans have knowledge that the universe is deterministic.”
  - roenxi 6y ago
    It means we can't be certain. Whether the universe is random or deterministic is, ironically, a matter of probabilities.
  - thisiszilff 6y ago
    I think this is related to the bayesian/frequentist approach to statistics. Bayesian statistics revolves around incorporating knowledge to update beliefs and probabilities express degrees of belief. IE, bayesian probability is the logic of uncertainty and how we should reason with imperfect information. The most famous example of this is the "draw a ball from an urn" example, and how we frequently day we "shake" the urn after we add the balls to "randomize" it. There is a lot of wordplay going on in stats to effectively forget certain information so that the problem I'd workable
zomglings 6y ago
The circularity argument for the frequentist interpretation of probabilities seems lazy to me. The essay argues that the frequentist view of probability is circular because it "reduces probability claims to probability claims". One can attempt to resolve this apparent circularity by thinking in terms of claims about the mathematical theory of probabilities versus claims about an empirical theory of probability (involving limiting behavior of experiments). Frequentist statistics could possibly be seen as a means of reconciliing these mathematical and empirical theories. The argument of the essay precludes this kind of interpretation of frequentist statistics without even considering it.
zaptheimpaler 6y ago
Kolmogorov complexity is a useful concept in the abstract but since it's not computable I find it hard to see anywhere it can really be applied. It doesn't solve the impossible problem of somehow deciding the true information content of any piece of data, it only locks it away in a slightly neater box of impossible. How much some data means depends completely on everything else you know about the world. You could imagine under different priors of knowledge, different strings would have differing kolmogorov complexity. not technically true, because kolmogorov complexity Is fixed, but that assumes you have the absolutely omniscient model for everything.
- EbTech 6y ago
  The reference by Rathmanner & Hutter presents a useful analogy. It argues that Kolmogorov complexity (and Solomonoff induction) are best viewed as a conceptual gold standard, like a perfect chess computer that does an exhaustive tree search. Practical methods are approximations. There are a few results where researchers were able to automatically infer evolutionary trees and such, by using a standard compression algorithm in place of K(x).
danielearwicker 6y ago
I think of probability as a summary of the structure/symmetries in a model. A model of a jar of red and blue beads is characterised by the ratio of red to blue, and that beads are only distinguished by colour.

220 points by EbTech 6y ago

Since this topic isn't so well-known, I wrote the case arguing that frequentist interpretations don't work, but algorithmic information theory (Kolmogorov complexity) does. I want to make this accessible and persuasive, so thoughts, questions, and arguments would be appreciated!

opheliate 6y ago
Such an interesting post, thank you for sharing! I'm in my second year of a maths degree currently, and we obviously studied frequentist probability/stats in the first year, but I'm not taking any probability modules this year. I found the tone & accessibility was just right for me :)
enriquto 6y ago
A simple sentence that I've found useful for pedagogy: "the probability of that coin toss being 50% does not talk about the coin; it talks about you, and about your partial knowledge of the universe." You can add: "The coin toss itself is deterministic and the result can be computed if you know the initial position and speed." They will inevitably bother you about the physical impossibility to measure the starting position and speed exactly, and then you say "ok, forget about the coin. You have 5 white and 5 black balls inside this opaque cylinder. What's the probability that the top ball is white? This does not talk about the balls (the color of the top one is already determined) but about your partial knowledge of them". (EDIT: formatting)
- alisonkisk 6y ago
  How does the ball situation help? It has the same problems of physical impossibility of measuring however the balls were ordered. (Modelling someone's brain?) I guess the argument works on someone with an unscientific model of the human brain, but that's one step forward and two steps back.
  - enriquto 6y ago
    Somebody just put the balls there carefully, and did not tell you in what order, just how many of each color.
    - Frost1x 6y ago
      That still boils down to a lack of prior information which I don't think removes the argument for "I don't have enough information." Probably have to use actual quantum phenomena that behave probabilistically by definition if you want a currently irrefutable physical example. I'm personally not convinced even this is fundamentally probabilistic and we currently have to rely on probability theory as a crutch for complex behaviors we just quite don't understand yet or don't have the time and resources to compute.
      - posterboy 6y ago
        You don't need quantum physics to formulate a philosophical standpoint that happens to agree with the Kopenhagen Interpretation. I'm not sure if it helps, but I suppose a compromise here would be the assumption that you don't really know the starting configuration of yourself, why you draw probabilistic inferences naturally, that the sun will go up tomorrow like every day. If that has a biologic explanation, then the top comment was not just to the illusive argument of platonic ideals.
      - enriquto 6y ago
        > "I don't have enough information." My point exactly. Probability theory is a precise mathematical formalization of the concept of "not enough information".
- st1x7 6y ago
  > the probability of that coin toss being 50% does not talk about the coin; it talks about you, and about your partial knowledge of the universe. But it does talk about the coin - a weighted coin would have a different probability. Same in the example with the white/black balls - if they weren't 5 white and 5 black but 6 white and 4 black, the probability you would assign to the top one would be different. Again, the probability is a way to describe the balls themselves, not just our knowledge. I get the general idea of representing probability as uncertainty and partial knowledge but your statements strike me as just straight up incorrect.
  - wearsshoes 6y ago
    This is still a partial knowledge situation - you have the information that there are a certain proportion of colored balls in the chamber, but not the information about their order. The probability includes the information we do know and allows inferences about information we don’t know.
    - st1x7 6y ago
      Sure, I just can't agree with the parent statement that probability has nothing to do with the object it describes, which is demonstrably false.
      - StavrosK 6y ago
        I don't think it's demonstrably false: If you don't know that the coin is weighted, the probability is 50%. Probabilities are predictions and estimates, not fundamentally about the thing itself, but about what we know about the thing.
        deleted 6y ago
        [deleted]
        mannykannot 6y ago
        The principle of indifference? I know that it is a commonplace assumption, but feels to me as though one is assuming one has more information than is justified. Coming back to the article's "economist's wager", is it rational to bet with even odds on something you know nothing about? If the assumption is interpreted as a testable hypothesis about outcomes, why would complete ignorance imply any particular result? On the other hand, if it is interpreted strictly as a statement about one's knowledge, why present it exactly as if one had sufficient knowledge of the situation to know that the probability is 0.5? Maybe the author will have an answer in part 2.
- Sharlin 6y ago
  This is basically the Bayesian interpretation of probability.
  - enriquto 6y ago
    Of course. But if you pronounce a fancy word like "bayesian" there's a large amount of minds that shut irremediably.
    - v64 6y ago
      That's also why we call it QBism [1] instead of quantum bayesianism [1] https://en.wikipedia.org/wiki/Quantum_Bayesianism https://en.wikipedia.org/wiki/Quantum_Bayesianism
  - lottin 6y ago
    Saying that "the probability of a coin toss of 50% talks about you" is not an interpretation of probability. Saying that we are "50% sure" is also not an interpretation of probability. It's a nonsensical statement. It's like saying we are "50% angry". It doesn't really mean anything.
    - alisonkisk 6y ago
      I don't understand your claims that these statements are are meaningless. They are commonly uttered and understood.
      - lottin 6y ago
        I can understand expressions such as "pretty sure" or "completely sure". I do not understand the expression "to be X% sure". If someone says they're "37% sure" tomorrow will rain, what does that mean exactly?
        dinosaurdynasty 6y ago
        37% of the time that someone says they are 37% sure of a statement X the statement X is true (assuming they're calibrated correctly/etc).
- nimbleal 6y ago
  It may be deterministic, but are you sure it would be computable? One does not necessarily imply the other.
- kps 6y ago
  Forget the coin and the balls — does the nucleus decay? You're not missing any knowledge; there isn't any.
  - jbay808 6y ago
    This is definitely the most interesting example, but it's not obvious that a situation where the relevant information is fundamentally inaccessible is a situation where you aren't missing any information. It's your best bet for a scenario where you can be sure that nobody else has more information than you do, though.
- mannykannot 6y ago
  Wigner's friend might have something to say about this... I don't have a specific argument to make here, only the feeling that if it were all just a matter of what a given observer knows, no-one would be talking about there being a QM measurement problem.
  - codethief 6y ago
    This is a very good point! In fact, some people do argue that there is no measurement problem in the Copenhagen formulation of quantum mechanics to begin with – at least if you take it seriously and strictly go by the rule that the laws laid down by Bohr et al. only concern you as the observer and your knowledge about the system, and not the system itself. Following this train of thought, there is nothing "real" about the wavefunction and it is just a tool to come up with predictions. The same goes for the collapse of the wave function (which just describes a change in your ability to predict future measurements, and not a change of the object) and the term "measurement" (which we might as well replace with "enlightenment", i.e. the moment in which we obtain knowledge about the system). In that sense, the only difference between classical and quantum mechanics is that our knowledge (viewed as a mathematical quantity) behaves differently in both theories: In classical physics, when we conduct multiple measurements of a given system in a row, our knowledge about that system will increase – to the point that, once we have measured all system properties to sufficient accuracy, we'll able to predict what any future measurement of any of those properties will yield (again, with some predictable uncertainty). So the knowledge of all our measurements has added up, it is an additive quantity. In QM, this is fundamentally different: We can only know anything about the object the very moment we look at it. The rules of quantum mechanics (again, in the very strict interpretation laid out above) dictate that the second we conduct a measurement, we can forget about any knowledge obtained through previous measurements of other (conjugate) observables: Future measurements of those observables are inherently unpredictable. In that sense, our knowledge about quantum-mechanical objects never "adds up" to anything. (To see that this is really the the distinguishing feature between classical and quantum mechanics, recall that the existence of conjugate observables really is the only thing setting apart the quantum from the classical world: Without conjugate observables it would be impossible to distinguish, say, 100 electrons in a superposition of spin up and down from an ensemble of 100 electrons of which 50 are in a spin up state and the other 50 are in a spin down state.) Of course, this whole interpretation is very unsatisfactory to lots of people (myself included) for a whole bunch of reasons. I assume that, to a large degree, this is due to the fact that laws of nature that put human observers in their very center seem rather undesirable. (At least since the time we switched from a geocentric to a heliocentric view of the world.) But my impression is that there's another reason: Our intuition from classical mechanics & statistics has taught us that objects exist independently of us as observers and behave in a deterministic fashion, at least provided we as observers know enough about them. (Meaning that the more we know about the coin's initial position and velocity, the more likely we are to predict the outcome of the coin toss. If we don't know anything about the coin, though, the outcome is as unpredictable as measuring spin up/down in quantum mechanics.) Unfortunately, this whole line of argument is circular: The reason we believe that the existence of physical objects is independent of us, is precisely because knowledge in classical mechanics is an additive quantity and we can get to the point where we know "enough" to come up with deterministic predictions. That is, we never have to discard knowledge when running new measurements and so our knowledge takes on a independent "role" – which we call reality.
- MereInterest 6y ago
  As a caveat, while this intuition works for classical mechanics, it does not work for quantum mechanics. All observations are consistent with wave function collapse being fundamentally random. Any hidden variables would need to be transmitted many times faster than the speed of light (~10000x, last time I checked the experiments), and are therefore inconsistent with our understanding of special relativity.
  - smallnamespace 6y ago
    Note that pilot wave theory, an (out of vogue) interpretation of quantum mechanics, also recasts the apparent randomness in quantum mechanics as due to our ignorance of the exact state of the pilot wave. Even Einstein struggled with quantum mechanics, famously saying "[God] does not play dice with the universe".
- JamisonM 6y ago
  There seems to be a lot of quibbling about the simple sentence here but I find it clarifying. Discussing coin tosses is a thought experiment with a very practical physical analog so spelling out clearly what the thought experiment's actual subject is has valuable properties so you don't get lost in the weeds of the physical execution of flipping coins.
- deleted 6y ago
  [deleted]
alisonkisk 6y ago
Why is frequentism bad because it only gives certainty for infinite samples, but complexity is good despite being non-computable? It's two sides of the same coin -- computable uncertainty va non-computable certainty.
- EbTech 6y ago
  I don't think frequentism is "bad"; just insufficient as a gold standard interpretation of probabilistic claims. I liked an analogy from the reference by Rathmanner & Hutter: the most "correct" chess-playing program involves a complete search along the tree of possible games. In practice, we try to approximate this ideal. In the case of Kolmogorov complexity, a reasonable takeaway might be to use the shortest program that we're able to find, even if it's not the shortest overall.
TheOtherHobbes 6y ago
OK - so apply Kolmogorov complexity to election polling. How does that work out? I think you're confusing various possible maps with the territory in a less than useful way. Given that frequentist interpretations are approximations - and understood as such - and Kolmogorov complexity isn't computable at all, what problem have you solved here?
- EbTech 6y ago
  Hm I admit it's hard to talk convincingly about election prediction, since we don't have practical algorithms to do this; a lot of it comes down to human judgment. The philosophical point (which might be approximated algorithmically someday, or by intelligent minds today) is that your election probabilities should come out of an overall highly compressed model of the world. In theory, a Bayesian who uses the prior 2^-K(x) over all strings x should, with sufficient life experience, come up with good estimates, in a certain sense. I'll have to think about this example more carefully when fulfilling my promise of writing about how this theory relates to everyday decision-making. Thanks for pointing out a potential weakness :)
qsort 6y ago
The article is excellent, congratulations. A couple observations/questions. 1) You didn't comment on the bayesian viewpoint that probability reflects a subjective idea about the state of the world. One might argue, for example, that probability isn't measurable, and that therefore, strictly speaking, a statement about the objective probability of an event isn't meaningful. Experimental evaluation would have to be done on an entire model instead. Do you have any objections to that point of view? 2) I don't find the case about Kolmogorov complexity to be actually convincing, at least not as per the requirements the rest of the article sets. "3141592..." could pass as either "random digits" or "first digits of pi". The fact that it's highly unlikely a true RNG would have generated exactly those, we are back to a frequentist argument there. It's likely I'm missing something, could you elaborate more or give me a pointer?
- SmooL 6y ago
  Isn't that what the occam's razor argument was for? Sure, both an RNG and the "40 digits of pie" program can produce that output, but the "40 digits of pie" program is shorter, therefore having less kolmogorov complexity
  - qsort 6y ago
    Yeah but is it? They are both programs with logarithmic Kolmogorov complexity, and Kolmogorov complexity is only defined up to constant factors unless you commit to a computational model. If you do commit to one, which is the shortest is just a function of what are the specifics of the model you chose, which isn't really interesting, it's literally code golf at that point.
    - FartyMcFarter 6y ago
      > They are both programs with logarithmic Kolmogorov complexity In the case of true random numbers, how is that so? Very few random sequences can be generated by a logarithmic-sized program, since most strings are not significantly compressible [1]. A simple counting argument shows that: there are 2^n strings of n bits, but only 2^(lg n) = n logarithmic-sized strings, a much smaller number! [1] http://theory.stanford.edu/~trevisan/cs154-12/kolcomplexity-rev.pdf http://theory.stanford.edu/~trevisan/cs154-12/kolcomplexity-...
JamisonM 6y ago
Some thoughts on the composition: * "I wrote the case arguing that frequentist interpretations don't work, but algorithmic information theory does": if that's what you are up to here then I think it would be for readers if you stated that up front in some way. And hit me with some kind of summary at the end that makes the concise version of your argument at the end, it's a long article. * Shorter might be better: There's a lot of stuff in here that I think you can pare out in the probability discussion that maybe isn't adding that much to your argument. I think there is a lot to be gained by assuming a generous reader. * Betting might be a distraction to your point: This might be confusing the imperfect knowledge of participants in a market with the imperfect knowledge of all the physical forces involved in a physical phenomenon and how that related to the seeming "randomness" of a coin flip for your reader. (The liquidity and stuff.. this is just not related to your point.) * Don't undermine your point with unrelated assumptions: "I imagine they wouldn’t consider their world unlikely at all: they would just add a new law to their description of physics: all dice, as if by divine intervention, are deemed to exhibit this strange behaviour" this lead me to think that you were just sort of shooing away the whole last X decades of high vs. low energy physics, we collectively certainly don't think that we have the rules correct precisely because of this complication, we find the idea that we need 2 sets of rules improbable and believe that there must be a way to explain everything with a single set of rules. So your mythical dice society probably would consider their dice exception a very unlikely world.. they would be confident they have the world wrong! A dubious assertion (or at least one that would need a whole lot of explanation) can be an off-ramp for a subset of readers.
- EbTech 6y ago
  Thanks for the detailed critique! I'll take some time to think about how to better make the points that I wanted to convey with those sections.
richard_todd 6y ago
I would have liked to know what the Kolmogorov approach has to say about the examples used to deflate frequentism ("which of my friends will start a business?"). I don't see from the article how the "smallest-program" approach could say anything useful about those, either. Maybe that wasn't the point--but after poking holes in the frequentist view, it uses unrelated examples like digits of pi to illustrate the Kolmogorov idea, so I'm left unable to directly compare the kinds of statements the two approaches can make. Even going back to dice or coins would have helped me compare them. Like, I know frequentists can show how the variance in coin-toss outcomes decreases as the sample size increases. What can the smallest-program approach say about that? Or was the point that those variant outcomes aren't "real" enough to talk about? Does that mean there is a connection to constructivism in mathematics here? It seems either approach benefits from more data, and there must be a concept related to a "confidence interval" where, as 100, then 200, then 300 digits of pi roll in, your pi-program stays the same size while other programs have to keep growing to accommodate the new data. Like, the ratio of the smallest program to the naive encoding ought to say something about how potentially predictive the small program is. Thanks for the interesting article. It definitely made me think about the issues, and now I'm curious to know more about the topic.
- EbTech 6y ago
  Thanks! I hope to better address your concerns in Part 2.
- PeterisP 6y ago
  One aspect of Kolmogorov approach is that implicitly models things like biased probabilities through compressibility. The shortest representation of rolls of fair dice or coins is their exact results, but if there's "less randomness" in some way (biased coin/die, sum of two dice which means non-uniform probabilities, combination of some predictable pattern with random noise) then there are more compact representations of that information, and all of that gets captured by the Kolmogorov approach without any explicit handling of the various possibilities.
analog31 6y ago
My only thought is that discovering a workable definition of "scientific method" is a reach. Philosophers spent a century searching for such a thing, in vain. On the other hand, providing something that just works would be beneficial enough, even if falling short of the philosophical holy grail, so it's worth pursuing. I'm a physicist, and physicists have always wondered why math works so well in physics. There's this famous essay by Eugene Wigner: https://www.dartmouth.edu/~matc/MathDrama/reading/Wigner.html https://www.dartmouth.edu/~matc/MathDrama/reading/Wigner.htm...
topsycatt 6y ago
There were a few points that I, as someone unfamiliar with many of the ideas presented, got hung up on. First, the paragraph that begins with "At first blush, the requirement to use..." Seems to be a non sequitur. I don't fully understand how the previous section creates a requirement to use deterministic programs, so I could use more explanation on how that requirement is established. Second, a very simple concrete example of what one of these programs would look like would be immensely helpful. After re-reading the article a bit I have a mental image of a program that contains a long, compressed string and a decompression algorithm that somehow models the system you're interested in. I can imagine how you might get a useful interpretation of probability from the decompression system, but there are enough open questions there that I'm not sure I have the correct interpretation. Hope that helps!
- EbTech 6y ago
  Thanks. I should clarify that the computer is deterministic, so as to avoid building randomness into the definition of randomness! I skimmed over an example too quickly, but your intuition is about right. For that sequence, two possible programs are: - Compute and print the first 40 digits of pi. - Decompress the following string according to a Shannon code with probabilities (1/36,1/18,1/12,[etc]): [insert code]
carapace 6y ago
This is awesome! Good work! Are you going to touch on Chaitin's Omega?
- EbTech 6y ago
  Thanks! :) I wasn't planning to go there! While I enjoy the idea, for now I'm trying to focus on what's needed to make sense of the problem of induction. Is there a nice connection that I missed?
danabo 6y ago
It sounds like you are arguing that i.i.d. frequentism doesn't work. I view AIT as generalizing frequentism to non-i.i.d. timeseries. This is formalized as Martin-Lof tests for randomness, and Solomonoff induction.

REALiSTiC 6y ago

Solid write up. If anyone's interesting in knowing more about how probability _came to be_ 'regarded' as real, i highly recommend Ian Hacking's "The Taming of Chance". Thought provoking in so many ways.

MauranKilom 6y ago

> The scientific method only works because the rules of the universe happen to be simple, while the set of observations it offers is vast. Kolmogorov complexity captures this defining characteristic of our reality. I've never seen this spelled out so beautifully!

willis936 6y ago
Not everything that is true is simple or catchy. Biasing towards it is a trap. PBS SpaceTime has a good discussion on this topic. https://youtu.be/xFKgIOX8IRE https://youtu.be/xFKgIOX8IRE
- jhardy54 6y ago
  Neat video, but it specifically disagrees with your point. He advocates for applying scientific rigor between leaps oc intuition. There's nothing wrong with GP appreciating beauty.
  - willis936 6y ago
    My point isn’t contradicted by the video. My point is not "beauty should not be appreciated". My point is that we should not dismiss models of nature based on aesthetics. Nature has no preference on what we see as beautiful. The video makes the point that beauty should be treated as a guiding principle, not as a hard and fast rule to sniff out truth.
karmakaze 6y ago
The patterns in our observations appear simple, the rules are deeper: e.g. newtonian motion vs relativistic.
deleted 6y ago
[deleted]
ordu 6y ago
It is a funny statement, I like it. But due to a different reason: all we know about reality is our theories. How we could state, that rules of the universe are simple? As I see it, we could state, that our theories are full of simple rules. But the universe have no theories nor rules outside of human's mind. It is completely our inventions, our dreams, our hopes that the universe have some rules. We could state that our simple rules works, but what does it mean "to work"? For example, a spider sees reality not like us, it feels vibrations of it's web, runs to a source of vibrations and start to bite, to wrap intruding object with web. It would do it to a tuning fork, if you pressed it to spider's web. His simple rules of reality works though. Despite the fact that sometimes spider bites steel of a tuning fork without any benefits for the spider. How could we know that our theories not just extended version of spider's? With the same issues, like they make us to do something absolutely pointless. How could we evaluate this fact? To ask our theories? But our theories already predicted that this pointless thing we do would be a good thing. We might ask our theories again and we'd get the same answer. This statement seems as a tautology for me. Our rules are simple, because they are simple. Our theories work because they tell us, that they work.

shrubble 6y ago

Probably...

rcthompson 6y ago

I'm glad to see a write-up of this. I've been searching recently for a way to reasonably define probability without having to invoke either hypothetical infinitely repeated experiments or placing bets (since the latter is really just implicitly invoking the former). I'm looking forward to part 2!

peterwoerner 6y ago
Check out the first chapter of Jayne's probability theory. It's the clearest take down of the frequentist interpretations that you have and introduction to the Bayesian interpretation. http://www.med.mcgill.ca/epidemiology/hanley/bios601/GaussianModel/JaynesProbabilityTheory.pdf http://www.med.mcgill.ca/epidemiology/hanley/bios601/Gaussia...
- rcthompson 6y ago
  Thanks, I had a quick skim through it and it seems really helpful. If I understand correctly, the claim is that the laws of probability as they are known comprise the only possible interpretation that satisfies the 3 desiderata set forth in chapter 1.
  - peterwoerner 6y ago
    I don't remember, the thing which really helped me click at least the bayesian interpretation was if we have A --> B and B, then A is more plausible and probability reasoning is a way to quantify how more plausible A is now that we know B is true. Chapter 5 is also a doozy in terms of explaining some of the things going on currently.

dandanua 6y ago

It's complex! (according to quantum mechanics)

xwdv 6y ago

Ultimately, probability must be real, because if you zoom in to a low enough level, the movement of particles is basically random, and this randomness is enough to poison any deterministic explanations for which side a coin flip will land. You could calculate which side a coin will land if you knew all the variables and thus arrive at a higher probability that it will land on a certain side, but it’s still a probability, because the particles that make up the coin could all move in such a way at any point that causes the coin to fall in a way you didn’t expect, even if that probability is very low. A small probability unlikely to happen is still a probability nonetheless. Try computing how a dice made up of a few particles will land. Good luck...

mellosouls 6y ago
I haven't read the article either but skimmed it briefly for a later read; I think it isn't disputing the phenomena that probability is ultimately used to describe (as you seem to claim), but how they are to be interpreted.
walleeee 6y ago
Philip McShane expressed this nicely: "...a thing is defined by... systematizations of coincidental aggregates of the properties of lower things". Probability theory "allows for the emergence of the systematic from the non-systematic".

thread_id 6y ago

It probably depends on the p-value.

FinanceAnon 6y ago

"All models are wrong, but some are useful"

deleted 6y ago

[deleted]

smitty1e 6y ago

> “Given competing programs whose outputs match our observations, always prefer the shortest.” I need to know context. Output is necessary, but not sufficient. Is this a stand-alone chat server for some niche voices, or, say, Twitter?

loup-vaillant 6y ago

Tow things. First, probability is real: it's a construct in one's mind, and minds are just as real as dice or coins. https://www.lesswrong.com/posts/f6ZLxEWaankRZ2Crv/probability-is-in-the-mind https://www.lesswrong.com/posts/f6ZLxEWaankRZ2Crv/probabilit... Second, (and the author may be leading up to this), there's Solomonoff's theory of inductive inference, which he has proven complete: when we apply Occam's razor (where the prior probability of each possible theory drops exponentially with its size), the amount of error a perfect Bayesian makes as they observe event and bet on the next one, ad infinitum, is finite. Roughly proportional to the complexity of the simplest theory that correctly predict the whole sequence of events. It's one of the most convincing proofs that Bayesian reasoning works. https://en.wikipedia.org/wiki/Solomonoff's_theory_of_inductive_inference https://en.wikipedia.org/wiki/Solomonoff's_theory_of_inducti... There's just a little snag. Perfect Bayesian reasoning is impossible to compute, so us mortals have to resort to approximations. Just as perfect certainty isn't possible, perfect reasoning is not attainable. Oh well.

smallnamespace 6y ago
> the amount of error a perfect Bayesian makes as they observe event and bet on the next one, ad infinitum, is finite. There's a little assumption that you're leaving out, namely that the Kolmogorov complexity of the data generating process is finite. From Wikipedia: > expected cumulative errors made by the predictions based on Solomonoff's induction are upper-bounded by the Kolmogorov complexity of the (stochastic) data generating process Whether the universe (or our observations of the universe) have finite complexity is very much an unresolved philosophical question.
- yters 6y ago
  If quantum is really random, then our universe has infinite Kolmogorov complexity.
  - fny 6y ago
    Is Newtonian mechanics real? Well, its not strictly real, but it's apparently real enough for engineering cars, planes, and rockets. So even if quantum is really random, I'd bet an unbiased coin will still land on heads with 50% probability every time.

cjkarr 6y ago

Maybe?

forrestthewoods 6y ago

The best part of this is that dice are known to not be fair. There’s a whole niche for people who want to buy fair dice. I can’t find it but I once saw a post that stacked ~20 d20 dice. The difference in height based on what number you picked to stack was shocking. The dice were incredibly non-uniform.

HelloNurse 6y ago

Mostly well argued, but there's some very loose language, such as calling "inconsistencies" mere practical relevance issues (such as nonideal markets, nonprobabilistic decisions, ignorance and the ensuing arbitrage, changes in probability values). A bad theory can be logically inconsistent, but it is not the case of probability theory and its competing interpretations.

jmount 6y ago

One way to deal with probability is: defer interpretation until after one has axiomatized calculation by measure theory. I have a quick video on the topic here: https://youtu.be/DnTTAd1TDyQ https://youtu.be/DnTTAd1TDyQ .

alisonkisk 6y ago
Measure theory is a powerful tool, but how does that answer the question of what probability is? Akin to physics, studying the wave equation doesn't tell you what the wave equation is in the real world.

karmakaze 6y ago

I don't think "first forty digits of pi" should be admissible. As it depends on an external definition that's not computed by the representation. I could come up with a mathematical definition for any prefix of digits, give it a name and say the 'first x digits of C'.

alisonkisk 6y ago
Complexity is defined by size over an entire class of objects, not a single one. Pi is reused many many times at constant cost, wile your C becomes C2, C3, C4,...
skybrian 6y ago
That's just because it's a loose description in English. It would actually be a short computer program that calculates pi, and instead of the first 40 digits you could ask for, say, a megabyte of data, or take the limit as the amount of data grows. Transmitting a program to compute pi would be shorter than the data needed by any compression algorithm that isn't somehow based on knowing the trick. The same trick could be used for any mathematically interesting number. The point is that incompressible random sequences exist that are not like that. You can't do better than transmitting the sequence itself.
- karmakaze 6y ago
  Yes the point was merely 'referring to an external' isn't a good example of a minimal size description.

emerged 6y ago

Since the lowest levels of our understanding of the universe are probabilistic, it seems reasonable to assume probability is the most real thing we currently know. It might be that will change some day, but at the moment everything we take to be non-probabilistic is just a simplification of underlying probabilities.

leephillips 6y ago

Nice article, on the whole, and usefully provocative. But I have misgivings about making these close connections between information theory and scientific theory-making. As everyone knows, information theory leaves out any notion of semantics, as it should. But the important thing about our theories of the world is that they have meaning to us. The scientist searches for something that makes sense of the world, not an algorithm for computing a series of numbers. The theories that we search for may not have the smallest Kolmogorov complexity; the criteria that they satisfy go a lot deeper.

texasbigdata 6y ago
That’s such a great phrase: usefully provocative. Thanks for that.

TaurenHunter 6y ago

I like to think that probability is the ratio unknown/known, information available divided by all possibilities. You know that a coin has 2 faces (known=2) and that if you toss it, 1 face will be up (unknown which=1). The ratio is more about something in the mind than intrinsic to the objects. That helped me understand why in the Monty Hall problem, when you switch doors, the probability of getting the prize increases.

alexpetralia 6y ago

> "Somehow, we must narrow down our hypotheses. Maybe you think that’s easy: only a few hypotheses describe plausible dice behavior; the rest are patently absurd! But now you’re relying on intuitive judgment, not a rigorous methodology." Maybe, as philosopher Robert Pirsig theorized in "Zen and the Art of Motorcycle Maintenance", you must rely on Quality!

jbeam 6y ago

Very interesting article that cuts to the core of many of the issues with pricing insurance products. While the phrase is hardly limited to the actuarial world, "all models are wrong, but some are useful" is definitely an extension of this. The fluid nature of probability is the center of the insurance universe. Probability is always a moving target in the insurance world. Indeed, if it weren't, there wouldn't be much of a need for actuaries. Much of actuarial training revolves around the idea of credibility -- how credible is your sample set, what alterations should you make to old data to make it relevant to today, and what data should you add to it as a complement in order to relieve the model of the biases inherent in your sample size. This is inherently Bayesian in it's approach. Where it truly gets interesting is that insurance companies are very cognizant of tail risk -- the 1-in-100, 1-in-250, 1-in-500 events that can cause insurer insolvency if not properly accounted for. You can survive a miscalculated loss trend within reasonable bounds, but if you haven't thought about the potential Cat 5 hurricane that hits Miami-Dade then you are going to have some very unhappy investors. When it comes to these types of events, you mostly need to be in the right ballpark. The order of magnitude matters more than the exact number -- albeit the exact number matters quite a bit for regulatory reasons. This type of calculation for property lines has largely been outsourced to the stochastic models developed by companies such as AIR and RMS. A sudden change in their models, which I think is likely after this record breaking hurricane season, can inflict capital pressure on the industry almost instantly.[1] There are some actuarial papers from around 50 years ago that discuss information entropy as another way to approach the issue of constructing probability models, but they never really caught on. It seems that is likely due to the lack of widespread computing power. I'm hoping these ideas can gain some steam now that we can construct some of these distributions from Python and R. [1] There is a fantastic article by Michael Lewis that describes this issue at great detail: https://www.nytimes.com/2007/08/26/magazine/26neworleans-t.html https://www.nytimes.com/2007/08/26/magazine/26neworleans-t.h...

sxzxs 6y ago
For me, it comes down to what kinds of these risky models are most interesting. Some can be interesting because of potential profit or minimizing loss (financial or actuarial) and others are inherently (theoretical physics). To add to your tail risk point - I wonder how many people foresaw the Venezuelan oil crisis way back when, or even less likely, the Saudi Arabian oil complex attack in 2019. And of course, the current situation we're in with CoVID that an entire university of forward thoughtful looking people didn't call until it was a week away. As an aside, do insurance companies significantly alter their policies when such a cat-5 hurricane is imminent? What preparations would they make in the face of that sort of event? Are you talking about chaos theory in the last paragraph? I'll read that article you linked in a bit and see what more I have to say, from skimming through it looks as though my question from the previous paragraph may be answered.

nieve 6y ago

Isn't declaring "the raison d’être of probability theory is to explain the decision-making of individuals facing uncertainty" basically a claim that the sole purpose of probability theory is an economist's approach? It's entirely possible that much more work in probability is done without a bet or payoff in sight than is done for the sake of decision-making models. It seems like basing epistemological arguments about all probability theory on a framing friendly to very specific groups of non-mathematicians.

glial 6y ago
What is a probability? Nearly always, a probability isn’t a statement about the world, it’s a statement about your knowledge and the information you have. Even for frequentist statisticians, equating probabilities with proportions of outcomes is an admission that you only have partial knowledge about the outcomes. Of course it also has mathematical structure and properties that may be interesting to people for their own sake. And there may be interesting things to say about quantum physics using probability, but I think a historian of mathematics would not claim that quantum physics was the driving force for the development of probability theory. Anyway, the “raison d’être” != every conceivable use.
alisonkisk 6y ago
Can you give an alternate definition for probability than "decision making under uncertainty?" I've never heard one. Sure there are mathematical real-analytic "theories of probability" but that's abstract analysis of measure, until it's applied to answer how... probable something is

kolbe 6y ago

I work with probably as it relates to financial markets, and my gripe with probably is at an even lower level. Even with dice, the fact of the matter is that you will roll dice with almost entirely Newtonian forces acting on it, and it will settle on a number. This is not unknown to physics. It’s just unknown to us. And because we don’t have the data and computational power to know with any certainty where the dice will land, we call it random. Everything I do is centered around creating probability distributions of where a stock will be in in the future. I don’t do this because it is fundamentally unknowable, but because I cannot access all of the data necessary to know. So, I’ve come to regard probably measure as an interesting and useful tool, but one that has no connection to the reality of existence.

EbTech 6y ago
Fair enough! I'd like to point out that the Kolmogorov complexity approach can make sense of subjective probability too. Since you lack precise enough information to predict the dice roll, the most compressed way to write down your observations will involve a Shannon-style code with your subjective probabilities. If you have enough information but not enough computational power, the resource-bounded variants of Kolmogorov complexity may be more applicable.
- jtsuken 6y ago
  Can you please define Shannon-style code? (Obviously, you don't have the shannon code, the compression method, in mind). Also, isn't Kolmogorov complexity uncomputable and you run into multiple "who shaves the barber" issues, when trying to determine it?
  - EbTech 6y ago
    The compression code can be specified first. If you have a lot of data, the specification will be negligible in length, compared to the code itself. Together, the specification and the Shannon code give an upper bound on the Kolmogorov compmlexity. If this is the shortest known program, we may consider the Shannon code probabilities as our "best explanation" of the data. You can also get posterior probabilities using a universal prior such as 2^-K(x), but of course, this can only be approximated in the limit of infinite runtime.

kempbellt 6y ago

The article probably has more value than my comment.

nolite 6y ago

its pretty complex

abdulhaq 6y ago

Surely you have to define reality before approaching this question?

Koshkin 6y ago
I wonder if it is even possible. In math, science, and philosophy some (basic) terms are necessarily left undefined and their meaning is either left open to interpretation or assumed to be evident based on common experience or convention.

ikeboy 6y ago

Probability is not real. Probability is subjective, it depends on what you know, and everyone has a different set of things they know. I flip a coin and look at it, then ask two other people for their probabilities. One of them knows the coin is biased towards head such that it's twice as likely to land heads than tails on any given flip. The other knows nothing about the coin. The first person guesses a 66% chance of heads. The second guesses a 50% chance of heads. I, having seen the coin, say it's a 0% chance of heads. None of these probabilities are wrong. They're all correct given the set of knowledge that person had. Probability is subjective.

xvedejas 6y ago
Your example suggests to me the opposite: Probability is a real and objective way to describe the information you have. "Garbage in, garbage out" still applies when you have no information.
- ikeboy 6y ago
  The distinction between objective and subjective collapses under your usage.
  - kgwgk 6y ago
    https://www.youtube.com/watch?v=X5cQcmAtjJ0 https://www.youtube.com/watch?v=X5cQcmAtjJ0
  - xvedejas 6y ago
    No, the distinction is clear. If your assigned probability is something that someone else can reproduce using the same steps given the same information, then it is objective (yet contextual). Your examples each have a clear reasoning behind the assigned probabilities, they're not just opinion-based assertions.
    - ikeboy 6y ago
      Bob has a very simple algorithm to output probabilities. He just answers 50/50 for any yes or no question. This is reproducible. Is this objective?

grandrew 6y ago

I have a question about how Kolmogorov complexity is related to "Human complexity" or complexity of understanding? E.g. a program may be very complex in Kolmogorov terms, like describing 1000 random numbers - easy to understand: you have a database of numbers, and a simple procedure that would scan through it. You can also imagine some real-world microservices-based program with a good architecture and a lot of code that handles all the exception cases of incoming data, all easily understandable. And now imagine an optimizing compiler for prolog programs. It may have much less code but the algorithm will be so complex that it might be impossible to fully understand its behaviour. E.g. fast-downward is a great example of such a program. So I'm wondering what does Kolmogorov complexity actually tell? Or does it tell anything useful in "real"-world?

jdc 6y ago
What is easy or useful for one person to understand is often much less so for another.
renewiltord 6y ago
Well, that's altering the 'language' that you're measuring the KC against. For instance, in the language that CS majors use the phrase "Kolmogorov Complexity" is sufficient to encode the concept of KC itself. And in the context of that language plus the concept of KC and adding in this entire thread, the letters KC themselves encode Kolmogorov Complexity in entirety. So the 'real world' version isn't so easy to tell in absolute terms like that because the languages can differ. But if you were thinking of it like a compressor/decompressor pair, then moving things into the language is like moving things into the compressor/decompressor. Naturally then you conclude that the "human complexity" depends greatly on the humans since any pair of humans creates a new universal description language that we can see as some base language plus the jargon that they are both familiar with.
- qayxc 6y ago
  > in the language that CS majors use the phrase "Kolmogorov Complexity" is sufficient to encode the concept of KC itself Wait, isn't that just conflating "language" with "knowledge"/"information"? The underlying assumption here is that the CS major has an association of a concept encoded by the letters "Kolmogorov Complexity". This is not universal, though, i.e. there's no computation that could derive the meaning behind these letters from the encoding alone. It's like claiming "620" is sufficient to encode Mozart's "Die Zauberflöte" ("The Magic Flute"), because in the language of a musician, the Köchel catalogue number along with the context would enable them to decode the full meaning. But in reality you would still have to look up the number and the score somewhere so it's not really an encoding but more of a pointer or index. I'd see any technical term that way, in that the term itself is not an encoding, but a key/index/identifier of a concept, not a full definition of the concept itself.

roenxi 6y ago

Nice article, there is an even easier interpretation though: The mathematics of probability defines a bunch of objects that do not exist, anywhere. Random variables, expectations, probabilities, maybe a few others. Then there are situations in the real world that look a lot like those objects from certain perspectives. It is a bit like Escher's Ascending and Descending - sometimes people make things in the real world that look like the infinite staircase when viewed from the right spot. Similarly, sometimes we find things that look probabilistic (dice rolls, coin flips) when viewed in ignorance of the sum totality of the entire universe. That is why there are a bunch of statistical tests that determine if outcomes are distinguishable from a hypothetical process generating that outcome. tl; dr; I like the article, but it seems to answer it's own headline with "yes" and it is much easier to answer it with "no". There is, philosophically speaking, nothing that we can guarantee looks random from all perspectives.

analog31 6y ago
Perhaps the most useful connection is that statistics lets us study what kind of math we can do, when we know something about a set, but not everything about it. And our experience with the world is like that too.
burrows 6y ago
> There is, philosophically speaking, nothing that we can guarantee looks random from all perspectives. Does this imply that the universe is deterministic? I believe it is false to claim “Humans have knowledge that the universe is deterministic.”
- roenxi 6y ago
  It means we can't be certain. Whether the universe is random or deterministic is, ironically, a matter of probabilities.
- thisiszilff 6y ago
  I think this is related to the bayesian/frequentist approach to statistics. Bayesian statistics revolves around incorporating knowledge to update beliefs and probabilities express degrees of belief. IE, bayesian probability is the logic of uncertainty and how we should reason with imperfect information. The most famous example of this is the "draw a ball from an urn" example, and how we frequently day we "shake" the urn after we add the balls to "randomize" it. There is a lot of wordplay going on in stats to effectively forget certain information so that the problem I'd workable

zomglings 6y ago

The circularity argument for the frequentist interpretation of probabilities seems lazy to me. The essay argues that the frequentist view of probability is circular because it "reduces probability claims to probability claims". One can attempt to resolve this apparent circularity by thinking in terms of claims about the mathematical theory of probabilities versus claims about an empirical theory of probability (involving limiting behavior of experiments). Frequentist statistics could possibly be seen as a means of reconciliing these mathematical and empirical theories. The argument of the essay precludes this kind of interpretation of frequentist statistics without even considering it.

zaptheimpaler 6y ago

Kolmogorov complexity is a useful concept in the abstract but since it's not computable I find it hard to see anywhere it can really be applied. It doesn't solve the impossible problem of somehow deciding the true information content of any piece of data, it only locks it away in a slightly neater box of impossible. How much some data means depends completely on everything else you know about the world. You could imagine under different priors of knowledge, different strings would have differing kolmogorov complexity. not technically true, because kolmogorov complexity Is fixed, but that assumes you have the absolutely omniscient model for everything.

EbTech 6y ago
The reference by Rathmanner & Hutter presents a useful analogy. It argues that Kolmogorov complexity (and Solomonoff induction) are best viewed as a conceptual gold standard, like a perfect chess computer that does an exhaustive tree search. Practical methods are approximations. There are a few results where researchers were able to automatically infer evolutionary trees and such, by using a standard compression algorithm in place of K(x).

danielearwicker 6y ago

I think of probability as a summary of the structure/symmetries in a model. A model of a jar of red and blue beads is characterised by the ratio of red to blue, and that beads are only distinguished by colour.