# How can statements of probability be verified?

Suppose I said, "There is a 50% chance of it raining tomorrow", and someone else said, "No, it is actually 60%". How can we know which person, if either, is correct? What I am really asking for is how to correctly assign probabilities to statements. Also, I would like some references that discuss this problem.

• The funny thing with weather firecasts is that when computers got better the predictions got worse. I was surprized when a prediction was made here (Holland) about snow in the morning in the weekend. It was in the middle of the week. And it started to snow in the morning that saturday. Jul 9, 2021 at 20:03
• In the case of weather you cant put earths next to each other and see on how many earths the prediction comes true, more or kess. On all earths though it will get warmere or cooler in the winter. 100%. Jul 9, 2021 at 20:06
• Probability assignments depend on state of knowledge and background assumptions, and can only be made in the context of a model. Different meteorological models can produce different assignments, so they can both be correct. See SEP, Philosophy of Statistics on how assignments are generated in bayesian statistics, for example, but Cross Validated SE might be a better place for this question. Jul 9, 2021 at 20:27
• A response in the spirit of Popper would be to say that predictions cannot be verified, but they can be falsified. A predictor that repeatedly makes wrong predictions can be judged to be unreliable, even if those predictions are probabilistic. One way of doing this is using a Brier score. en.wikipedia.org/wiki/Brier_score Even then, we can speak only of the reliability of a predictor of a set of predictions, not of a single prediction. Jul 9, 2021 at 21:27
• See the following related question: philosophy.stackexchange.com/questions/32135/… Jul 13, 2021 at 11:03

[A long answer which got accepted but turned out to be basically wrong. Please see the edit history for details]

• The frequentist interpretation doesn't say that probabilities don't apply to the caught coin. That's like card games, like poker, where you randomize at the start and then the players' hands are hidden. The frequentist interpretation is able to analyze poker. As long as a situation can be set up repeatably, and results observed, then the frequentist interpretation can assign probabilities based on the long-run ratios. Jul 10, 2021 at 10:49
• I agree with @causative -- the frequentist interpretation doesn't require a distinction between events that have already happened and events that haven't, it just requires an objective truth about frequencies of different outcomes (in 'hypothetical frequentism' which I think is most common, these would be the frequencies in the limit of an infinite number of trials). Then if you have some known facts A about a trial (like the conditions under which the coin was flipped), and unknown facts B (what side came up), P(B|A) is the frequency of B in the set of trials with characteristics A. Jul 10, 2021 at 16:51
Jul 12, 2021 at 9:07
• Having read the comments and done some more reading I see that I was wrong. However I can't delete an accepted answer. Please remove the acceptance and I'll kill it. Jul 12, 2021 at 16:07

It's difficult to verify an individual probabilistic statement about a real-world event, especially if the estimate is as vague as 50%, but we can check large sets of probabilistic statements against observations. We can ask, when this person says the chances are 50%, what proportion of times does the event he predicted actually happen? If the event happens 75% of the time that he says it's 50%, then he's been (partially) falsified; not as severely falsified as if the event happened 5% of the time or 95% of the time. Falsification of probabilistic statements is a matter of degree and repetition.

Being a little more sophisticated we can look at the surprisal of the events, according to the person's predictions. If he predicts a 75% chance of rain, and it does rain, the surprisal is -log_2(0.75) = 0.41. If he predicts a 75% chance of rain and it doesn't rain, the surprisal is -log_2(0.25) = 2.0. The higher the surprisal, the less accurate he was. We can look at his average surprisal across many predictions.

For example, let's look at the person's rain predictions over 10 successive days, along with whether it rained (R) or did not (N):

``````10% 20% 5% 75% 90% 10% 5% 5% 5% 5%
R   N  N   R   R   N  N  N  N  N
``````

The average surprisal here is (-log(.1) - log(.8) - log(.95) - log(.75) - log(.9) - log(.9) - log(.95) - log(.95) - log(.95) - log(.95) ) / 10 = 0.47. This is a measure of how many bits, on average, we would need to correct his predictions to what actually happened. Higher is worse, 0 is the best you could do.

Another way to falsify probabilistic predictions is through certain forms of gambling. If you actually know the odds, and make decisions based on that, you will have a long-run advantage when you are gambling compared to someone who doesn't know the odds. The theory of probability was originally designed to assist in gambling. This is the principle behind prediction markets. If someone playing prediction markets loses in the long run, then his estimates of the probabilities of different events must have been inaccurate - at least when compared to his competitors.

Speaking more generally, when you're playing a game where you make probabilistic predictions and your reward is based on a proper scoring rule, your success in this game is maximized when your predictions are maximally accurate, and poor success in this game falsifies your predictions.

I've been describing ways to check probabilistic predictions against observations. But there are other ways to validate or falsify predictions, that don't require us to wait for the observation.

First, we can look at internal consistency. If someone says P(A) = 0.5 and P(B) = 0.7 and P(A ∩ B) = 0.8, then he has committed the conjunction fallacy, falsifying his assignment of probabilities.

Finally, we can look at the method that generated the probabilities, and the track record or theoretical guarantees of this method. Producing and evaluating such methods is basically all of statistics. If the method is good, then we should trust the probabilities more. If the method is untested or known to be bad, then we should trust the probabilities less.

• If the rain predictor is making his predictions based on a specific set of well-defined observations (numerical parameters whose values are found by observation, say), a frequentist can also assess more specifically whether the probability is correct in frequentist terms by taking a large set of trials (approaching infinity in hypothetical frequentism) and looking only at the subset of trials where those exact same observations were made, and then seeing in what fraction of that subset it did indeed rain. It could be his probability est. is accurate under those conditions but not in others. Jul 10, 2021 at 16:56

There are many interpretations of probability, as other answers have pointed out (though those answers are filled with errors and inaccuracies, so beware).

The most basic issue to point out, I think, is that your question assumes that utterances like "There is a 50% chance of it raining tomorrow" have a truth value. That is, you assume that it makes sense to say that such utterances are correct (true) or incorrect (false).

Bayesians (subjectivists, personalists) deny this. If Bob says, "There is a 50% chance of it raining tomorrow," that should be interpreted as "I (Bob) am as confident that it will rain as I am that it won't." If Charlie counters, "No, there is a 60% chance of rain," it doesn't make sense to ask who is correct. Bob and Charlie just have different opinions, different levels of confidence.

For more on this view, you'll want to read de Finetti's Theory of Probability and Isaac Levi's Enterprise of Knowledge. Or start with this.

• It's a good point that truth value gets more complicated when we use probabilities. But I think it goes too far to say that Bob's 50% prediction is nothing more than his level of confidence. He's at least claiming that it would be reasonable to judge the probability at 50%, given the available information to him. Consider this: if Bob says it's 99.999999999% likely to rain, and it doesn't rain, would you still say he wasn't wrong? Jul 12, 2021 at 9:34
• We have certain mental methods, both formal methods and intuitive judgments, that we use to arrive at probabilities. We want the probabilities the methods produce to generally match the observed frequencies, and help guide us to profitable decisions. If the methods are poor at helping us achieve those goals, then we can say the methods were wrong, and replace them with better methods if possible. We can also say an individual probability is wrong if our best methods would give a substantially different probability. Jul 12, 2021 at 9:42
• @causative All this sounds like frequentist question begging to me. I detect no argument against the Bayesian interpretation (which I am merely reporting and not defending).
Jul 12, 2021 at 10:02
• I think very few Bayesians would agree that a probabilistic statement can't be wrong as long as it is an accurate self-report of one's degree of belief. Bayesian probability is highly concerned with rational methods for updating beliefs, often formal mathematical methods, obeying Bayes' law. It's not just "anything goes." Jul 12, 2021 at 10:07
• @causative Many Bayesians do see the choice of the prior distribution as an arbitrary subjective matter, though some like Jaynes have argued there is a rational procedure for choosing your priors. Jul 12, 2021 at 14:44

A weather prediction is a conclusion from evidence. For a complete description of the problem, you need to include the evidence, which is basically all of the data that the weather bureau uses to make the prediction.

From the frequentist point of view, this evidence can be seen as the experiment; the rain or lack of rain the next day is the outcome. So suppose your evidence is a chart of barometer readings and wind velocity readings through the day and you make your prediction of 60% chance of rain based on that evidence alone. To a frequentist, the prediction is true if and only if 60% of the time that you have those exact readings, it rains the next day and 40% of the time it does not.

Of course you are never going to see those exact readings again, so you can't really measure your accuracy that way; instead we can write a function that maps a chart of barometer readings and wind velocity readings into a prediction of rain the next day, then we measure whether it rains 10% of the time that the model predicts a 10% chance of rain, 20% of the time that the model predicts a 20% chance of rain, etc.

Statements like the chance of rain tomorrow in the north of the country being 60% cannot be verified. It will rain or it will not. It could be a coincidence that the weather man just stated a 99% chance on rain in the north. If it was sunny for a whole week already and if it indeed rained in the north the next day he would be considered a miracle weatherman. But to keep up that status would be very difficult.

How can you ever verify that the chance was indeed 50% or 60%. Which weatherman can be trusted rhe most will depend on this. The only thing to be done is let a number of Earths develop with slightly different initial cinditions and look how many of them show what kind of weather patters. How will you ever do this in reality? By making approximations of the current weather and let the variations in the abstract weather pattern evolve in a computer. For every different initial pattern a different pattern will emerge. The chance on rain is the number of different patterns with rain in the north divided by the total number of patterns.

There are various levels of approximations and different schemes of calculations. Which one is right is hard to tell. It is gonna rain or not. Different approximations and different theoretical calculations give a different probability. Which one is the right one can only be known in the way I alredy mentioned. But that way is impossible to realize *contrary to throwing a dice).

You can think the chances are plausible by looking at earlier predictions of the weatherman. If he predicted correctly in the past then then you can be sure he won't just tell you something. By a 59% chance he means that he just don't know (when taking computers into consideration only). 90% chance you can take your umbrella with you. If you trust her.

If you trust another way of predicting more then you should listen to the other kinds of prediction. I can predict that next winter the temperature will be lower than now (measured over a week and taking the mean). It is for 99% true. I can even oredict that the mean temperature on Earth will rise in the next 25 years. It takes no computer for that. And I listen to the Hopi when they say that a huge cloud will rain hot ashes when temperature rises and the face of the planet is changed too much by people.