Suppose I said, "There is a 50% chance of it raining tomorrow", and someone else said, "No, it is actually 60%". How can we know which person, if either, is correct? What I am really asking for is how to correctly assign probabilities to statements. Also, I would like some references that discuss this problem.
It's difficult to verify an individual probabilistic statement about a real-world event, especially if the estimate is as vague as 50%, but we can check large sets of probabilistic statements against observations. We can ask, when this person says the chances are 50%, what proportion of times does the event he predicted actually happen? If the event happens 75% of the time that he says it's 50%, then he's been (partially) falsified; not as severely falsified as if the event happened 5% of the time or 95% of the time. Falsification of probabilistic statements is a matter of degree and repetition.
Being a little more sophisticated we can look at the surprisal of the events, according to the person's predictions. If he predicts a 75% chance of rain, and it does rain, the surprisal is -log_2(0.75) = 0.41. If he predicts a 75% chance of rain and it doesn't rain, the surprisal is -log_2(0.25) = 2.0. The higher the surprisal, the less accurate he was. We can look at his average surprisal across many predictions.
For example, let's look at the person's rain predictions over 10 successive days, along with whether it rained (R) or did not (N):
10% 20% 5% 75% 90% 10% 5% 5% 5% 5%
R N N R R N N N N N
The average surprisal here is (-log(.1) - log(.8) - log(.95) - log(.75) - log(.9) - log(.9) - log(.95) - log(.95) - log(.95) - log(.95) ) / 10 = 0.47. This is a measure of how many bits, on average, we would need to correct his predictions to what actually happened. Higher is worse, 0 is the best you could do.
Another way to falsify probabilistic predictions is through certain forms of gambling. If you actually know the odds, and make decisions based on that, you will have a long-run advantage when you are gambling compared to someone who doesn't know the odds. The theory of probability was originally designed to assist in gambling. This is the principle behind prediction markets. If someone playing prediction markets loses in the long run, then his estimates of the probabilities of different events must have been inaccurate - at least when compared to his competitors.
Speaking more generally, when you're playing a game where you make probabilistic predictions and your reward is based on a proper scoring rule, your success in this game is maximized when your predictions are maximally accurate, and poor success in this game falsifies your predictions.
I've been describing ways to check probabilistic predictions against observations. But there are other ways to validate or falsify predictions, that don't require us to wait for the observation.
First, we can look at internal consistency. If someone says P(A) = 0.5 and P(B) = 0.7 and P(A ∩ B) = 0.8, then he has committed the conjunction fallacy, falsifying his assignment of probabilities.
Finally, we can look at the method that generated the probabilities, and the track record or theoretical guarantees of this method. Producing and evaluating such methods is basically all of statistics. If the method is good, then we should trust the probabilities more. If the method is untested or known to be bad, then we should trust the probabilities less.
There are many interpretations of probability, as other answers have pointed out (though those answers are filled with errors and inaccuracies, so beware).
The most basic issue to point out, I think, is that your question assumes that utterances like "There is a 50% chance of it raining tomorrow" have a truth value. That is, you assume that it makes sense to say that such utterances are correct (true) or incorrect (false).
Bayesians (subjectivists, personalists) deny this. If Bob says, "There is a 50% chance of it raining tomorrow," that should be interpreted as "I (Bob) am as confident that it will rain as I am that it won't." If Charlie counters, "No, there is a 60% chance of rain," it doesn't make sense to ask who is correct. Bob and Charlie just have different opinions, different levels of confidence.
For more on this view, you'll want to read de Finetti's Theory of Probability and Isaac Levi's Enterprise of Knowledge. Or start with this.
A weather prediction is a conclusion from evidence. For a complete description of the problem, you need to include the evidence, which is basically all of the data that the weather bureau uses to make the prediction.
From the frequentist point of view, this evidence can be seen as the experiment; the rain or lack of rain the next day is the outcome. So suppose your evidence is a chart of barometer readings and wind velocity readings through the day and you make your prediction of 60% chance of rain based on that evidence alone. To a frequentist, the prediction is true if and only if 60% of the time that you have those exact readings, it rains the next day and 40% of the time it does not.
Of course you are never going to see those exact readings again, so you can't really measure your accuracy that way; instead we can write a function that maps a chart of barometer readings and wind velocity readings into a prediction of rain the next day, then we measure whether it rains 10% of the time that the model predicts a 10% chance of rain, 20% of the time that the model predicts a 20% chance of rain, etc.
Statements like the chance of rain tomorrow in the north of the country being 60% cannot be verified. It will rain or it will not. It could be a coincidence that the weather man just stated a 99% chance on rain in the north. If it was sunny for a whole week already and if it indeed rained in the north the next day he would be considered a miracle weatherman. But to keep up that status would be very difficult.
How can you ever verify that the chance was indeed 50% or 60%. Which weatherman can be trusted rhe most will depend on this. The only thing to be done is let a number of Earths develop with slightly different initial cinditions and look how many of them show what kind of weather patters. How will you ever do this in reality? By making approximations of the current weather and let the variations in the abstract weather pattern evolve in a computer. For every different initial pattern a different pattern will emerge. The chance on rain is the number of different patterns with rain in the north divided by the total number of patterns.
There are various levels of approximations and different schemes of calculations. Which one is right is hard to tell. It is gonna rain or not. Different approximations and different theoretical calculations give a different probability. Which one is the right one can only be known in the way I alredy mentioned. But that way is impossible to realize *contrary to throwing a dice).
You can think the chances are plausible by looking at earlier predictions of the weatherman. If he predicted correctly in the past then then you can be sure he won't just tell you something. By a 59% chance he means that he just don't know (when taking computers into consideration only). 90% chance you can take your umbrella with you. If you trust her.
If you trust another way of predicting more then you should listen to the other kinds of prediction. I can predict that next winter the temperature will be lower than now (measured over a week and taking the mean). It is for 99% true. I can even oredict that the mean temperature on Earth will rise in the next 25 years. It takes no computer for that. And I listen to the Hopi when they say that a huge cloud will rain hot ashes when temperature rises and the face of the planet is changed too much by people.