I'm going to use some extracts from an article by Matthew H. Haber to explain the difference between possibility and probability. These extracts set out (some of the) different kinds of possibility; and with regard to probability, give a rough first indication of the (or a) distinction between objective and subjective probability. Haber also takes account of gambling and knowledge. I've added further references. I never normally answer by quotation but Haber's article is clear, compact, and informative: I hope it helps. It won't if you reject the idea or relevance of laws.
Possibility
How should we understand
what is meant by "possibility" ...? Talk about possibility
is talk about what is and what is not ruled out. But
not ruled out by what? By what we know - epistemic
possibility; by the laws of physics - physical possibility;
by the laws of biology - biological possibility; and by
the laws of logic - logical possibility. Underlying the
above dialogue seems to be a concern over metaphysical
possibility. Metaphysical possibility is that which is not
ruled out by necessity; something is metaphysically
possible just in case it is not necessarily not possible
(Kripke, 1980; Jubien, 1997) (Matthew H. Haber, 'On Probability and Systematics: Possibility, Probability, and Phylogenetic Inference', Systematic Biology, Vol. 54, No. 5 (Oct., 2005), pp. 831-841: 833.)
Probability
Similar to the distinction made above between
metaphysical and scientific possibility, so too is there
a distinction between objective (or metaphysical) and
subjective (or epistemic) interpretations of probability.
Objective interpretations of probability are those that
take probability to be a thing of the world that exists independent of us. Subjective interpretations of probability,
on the other hand, take probabilities to be reflections of
degrees of belief about a proposition of some event or object of the world. So subjective probabilities, then, do not
exist in the world independently of our beliefs. A brief
example can help draw out the importance of making
this distinction.
Suppose, for example, that I had a coin that was known
to be biased, though the direction of that bias was unknown. Suppose, too, that I asked both an objective and
a subjective probabilist what the probability was that the
coin would land "heads" upon flipping. The objective
probabilist might respond with something like "if by
'probability' you mean objective probability, then all I
can say of the biased coin is that the probability of that
coin landing heads is not 50%. The actual objective probability of the coin
landing heads is something that we can
discover upon experiment and observation; but, given
that the coin is biased, we know the probability cannot
be 50%." The subjective probabilist, on the other hand,
might respond to the same question as follows, "if by
'probability' you mean subjective probability, I have no
reason for believing that the coin is biased either towards
heads or tails, so the only justified degree of belief is that
it is equally likely to be biased in either direction, and,
thus, I can contingently assign a 50% probability to the
proposition that the coin will land heads. Upon experiment and observation, we will be justified in adjusting
our degree of belief accordingly." So if one is not careful
to be precise about what kind of interpretation of probability is being discussed, there is great danger of mischaracterizing assignments of probability and confusing
the issues at hand. In the example just given, both parties
were right to gently chastise my ambiguous phrasing of
the question, as the divergent answers given turned on
which classification of probability was being assumed. (Haber: 833-4.)
Bayesian probability, mentioned in your question, is a version of subjective probability.
Gambling and knowledge
Take, for example, the gambler's fallacy: Roberto Alomar is batting
0.300. He comes to bat three times in a game and fails to get a hit.
... Our objective probabilist ... asserts that, because he is batting 0.300, he still has only a 30%
chance of getting a hit, but this ... fails to take into account the full
scope of knowledge. In the first place, because Alomar failed to get
a hit in his last three times at bat, he is actually batting 0.297; the
probabilities have changed, because they are historically contingent
phenomena. More to the point, Alomar either will or he will not get a hit
and there is no probability that can be assigned to that one event: betting
on one event alone is foolish. (Siddall, M. E., and A. G. Kluge. 1997. 'Probabilism and phylogenetic
inference', Cladistics 13: 332.)
References
Haber, M.H. 'On Probability and Systematics: Possibility, Probability, and Phylogenetic Inference', Systematic Biology, Vol. 54, No. 5 (Oct., 2005), pp. 831-841.
Jubien, M. 1997. Contemporary metaphysics: An Introduction. Blackwell Publishers, Malden, Mass.
Kripke, S. A. 1980. Naming and necessity. Harvard University Press, Cambridge, Mass.
Siddall, M. E., and A. G. Kluge. 1997. 'Probabilism and phylogenetic
inference', Cladistics 13: 313-336.