There are two things we mean by "probability". The first is propensity, and this has clear links to information theory, as another person commented about alread

There are two things we mean by "probability". The first is propensity, and this has clear links to information theory, as another person commented about already. It's important to emphasise that in terms of propensity, it doesn't matter whether or not the event has occurred, what matters is your knowledge about it. A flipped fair coin has a definite side up (as can be verified by a silent third observer) but for you, who has not yet observed which side it is, your best guess is still either side with 50 % probability. Similarly, if you only know there's a soccer game going on, you might guess that the stronger team will win with 60 % probability (based on historic frequencies of exchangeable situations), but someone who has seen the score and knows the weaker team has a lead and knows there's only a few minutes left of the game will judge there to be a 2 % probability the stronger team wins. Same situation, different information, different judgements. That's the first meaning of "probability". What we also mean with that word is "the rules of probabilistic calculation". These are based on mathematical ideas like coherency (if one of two things can happen, their probabilities should add up to 100 %) and can definitely be taken as axiomatic. All of this is not an answer to your question, but it might make the discussion richer.