3

In probability theory, there are two primary approaches to interpreting events: the frequentist perspective and the Bayesian perspective.

From a frequentist standpoint, each spin of a fair roulette wheel is an independent event with a 50% chance of landing on black. According to this view, no matter how many consecutive black outcomes are observed, the probability of the next spin being black remains 50%. This principle helps avoid the "Gambler's Fallacy," where one might incorrectly believe that past events influence future independent events.

On the other hand, the Bayesian approach involves updating our beliefs based on new evidence. If we observe a series of black outcomes, a Bayesian would consider the extremely low probability of such a sequence occurring by chance if the wheel were fair. Bayes' Theorem allows us to update the probability of the wheel being biased based on the observed data. The more consecutive black outcomes we observe, the stronger the evidence that the wheel might be biased.

Given these two perspectives, I have a question:

At what point (i.e., after how many consecutive black spins) should we reasonably start to suspect that the roulette wheel might be biased and thus incline towards a Bayesian approach? How do we determine this threshold, and what factors should we consider when making this decision? Specifically, how do we balance the theoretical independence of events with the practical improbability of long sequences of identical outcomes in a fair game?

7
  • 8
    I don't really understand why you're asking this question here. I think this is a purely technical (+practical) question about probability theory and statistics. Have you tried first asking the question on a math or statistics forum? Otherwise - how is philosophy involved here?
    – mudskipper
    Commented Jul 21 at 16:39
  • 2
    @mudskipper, on further consideration, I think there is philosophical content here beyond the technical statistics - the differences in approach stem from differences in the definition of probability, which doesn't seem that well appreciated on this SE, judging by some of the questions and answers.
    – user6527
    Commented Jul 22 at 14:02
  • 1
    @DikranMarsupial - I agree with your statement - I just did not really see that play out in the OPs question - probably because of the way it is expressed. I am fully +1 on your answer below - very precise criticism + correction on the question.
    – mudskipper
    Commented Jul 22 at 14:36
  • 1
    There are statistical tests to determine fairness. The chi-squared goodness of fit is an example. Commented Jul 25 at 15:42
  • 3
    This question is similar to: How improbable does an event have to be before we can say it didn't happen by chance?. If you believe it’s different, please edit the question, make it clear how it’s different and/or how the answers on that question are not helpful for your problem. Commented Oct 25 at 16:07

4 Answers 4

8

The question is a bit confused in it's use of statistical terminology and ideas, but I think there is an answer, so I'll address the terminology as I go:

From a frequentist standpoint, each spin of a fair roulette wheel is an independent event with a 50% chance of landing on black.

A Bayesian would also view it as an independent event. If events are independent, it just means the probability of one event does not depend on another event. As others have pointed out, saying that it is a "fair"" roulette wheel is essentially asserting that it has a probability of 50% of landing on black (I'm assuming there isn't a green zero slot). But that is what we are testing, so a frequentist would only assume it is fair to set up a "Null hypothesis".

According to this view, no matter how many consecutive black outcomes are observed, the probability of the next spin being black remains 50%

Again, a Bayesian would take that view as well.

On the other hand, the Bayesian approach involves updating our beliefs based on new evidence.

I think this is a misunderstanding of Bayesianism, taken in conjunction with the statement about frequentists in particular viewing the events as independent. In this scenario, a Bayesian would also model the events as independent, with a fixed, but unknown probability of landing on black. When we get data, the probability that the roulette wheel lands on black doesn't change (it is fixed by the physics of the wheel), what changes is our state of knowledge (a.k.a. belief) regarding the value of that probability.

Before getting to the methods:

Specifically, how do we balance the theoretical independence of events with the practical improbability of long sequences of identical outcomes in a fair game?

This is a misunderstanding, and can't be answered. Both Bayesians and frequentists would treat the events as independent, so there is nothing to balance. In both Bayesian and frquentist approaches there is a fixed probability of landing heads, but we don't know what it is.

There are various ways this problem can be approached, but the frequentist is limited by defining probabilities only in terms of a long run frequency. This means they can't assign a probability to the truth of a particular proposition, such as "the wheel is fair" or "the probability of landing black lies in some particular interval" - these are either true or false, and have no non-trivial long run frequency (i.e. other than zero or one). The Bayesian on the other hand, can assign a probability to a particular event the truth of a proposition, so they can give a more direct answer.

FREQUENTIST

What a frequentist would probably do is to use a Null Hypothesis Statistical Test (NHST). They would begin by setting out their null hypothesis (the hypothesis that must be nullified in order to claim that the wheel is rigged). In this case H0: p(black) = p(red) = 0.5. They would then evaluate the p-value, which is the probability of a sequence of events at least as extreme as that actually observed. To keep it simple, lets assume we observed the wheel landing on black five consecutive times. Under H0:

p = 0.5^5 = 0.03125

If this value was below the significance level, alpha, then the frequentist would say "the null hypothesis can be rejected" and then act as if the wheel was rigged.

The tricky bit here is working out the appropriate "significance level". People often opt for 95% level (expressed as 1-alph), but Gigerenizer views this as "Mindless statistics" and part of the "Null ritual" because it is symptomatic of the common use of NHSTs without a solid understanding of the frequentist framework, which commonly leads to misunderstandings. This is well illustrated by the XKCD cartoon

XKCD frequentists-v-bayesians cartoon

The error the frequentist is making here is to use a significance level that is far too lax, and a much smaller value of alpha is appropriate. The probability that the sun has actually gone nova is vanishingly small, so it would be unreasonable to continue with that hypothesis because the alarm would have a 1-in-20 probability of going off if it was due to chance!

Another tricky issue here, when performing an NHST, we say "we can reject H0" or "we fail to reject H0". This rather clunky wording is highlighting the fact that rejecting H0 does not mean we "accept" H1 - in this case, it is not evidence that the sun has gone nova. The reason we can't say that is because the test does not involve the prior probabilities or the likelihood of the observation if it were caused by H1. NHSTs are best viewed as being like Popperian falsification. The best we can say about H1 is that it has survived an attempt at falsification, however the severity of the attempt may be pretty low (as in this case) and so the corroboration of H1 may be rather slim!

So how should we choose the significance level? Many frequentists are not going to like this, but the truth is that it is a subjective choice of the statistician, based on their prior knowledge of the system, often with no straightforward means of calculating what it should be. The inventor of significance testing, Sir Ronald Fisher, wrote:

… no scientific worker has a fixed level of significance at which from year to year, and in all circumstances, he rejects hypotheses; he rather gives his mind to each particular case in the light of his evidence and his ideas.

The really awkward bit is that you can't make this choice by assigning probabilities to your prior beliefs and working out the significance level, because frequentists fundamentally cannot assign a probability to the truth of a proposition. This also means that there generally isn't a straightforward means of calculating what the significance level should be for some particular purpose, and the statistician will generally opt for a value that "seems about right". Frequentism is often viewed as being objective, but this isn't true, it still contains subjective elements and prior beliefs, but they are often ignored and (often deeply inappropriate) default values used instead.

Now, that would be a reasonable answer if you just looked at one set of five spins of the wheel. If you waited for five consecutive spins that landed black, then wait long enough and it will happen, even if the wheel is fair. To get around that, you could introduce corrections for the implicit "multiple hypothesis testing" (i.e. you are performing many tests, not just one). It gets complicated, but the significance level remains arbitrary subjective choice, with no clear method of calculating it.

Another issue is that no real roulette table will be exactly fair. It will always have some asymmetry such that the probability of landing on black is not exactly 0.5. This means in practice, if you collect enough data, you will always be able to show that the wheel is rigged, even if it is the fairest wheel ever constructed. This is because the H0 is based on a point value from a continuous distribution, so the probability of observing a particular point value is zero. You could get around this by defining H0 using an interval centered on 0.5, but that makes calculating the p value more difficult and probably not worth it. If it takes 1,000,000 spins to detect the bias, common sense should tell you that it is of no practical significance, even if it is of "statistical significance".

@DoubleKnot raises a very good point in the comments. If we observe a sequence of spins that all land on black, then we can just monitor the p-value to see when it gets a low enough value that we should be suspicious of bias. However, another way to look at the question is to ask how many spins of the wheel do we have to observe in order to be confident that if the null hypothesis was false that we would confidently expect the test to have rejected it. This is a sample size calculation, and it can be based on an analysis of statistical power. The power of a test is the probability of rejecting the null hypothesis if it is actually false, and it is usually denoted as beta. This is something that users of NHSTs rarely consider, which is part of the "Null Ritual" mentioned earlier. One of the tricky aspects in this case is that the power of the test depends on how rigged the wheel is. If it always lands on black, you should be able to reject H0 fairly quickly. If it is only rigged by a few percentage points (which would give a healthy profit for the casino in the long run) then you would need much more data. This makes the calculation more tricky. I have a blog post, "What does Statistically Significant Actually Mean?", at SkepticalScience, which contains a worked example for flipping a coin, which is a very similar problem.

BAYESIAN

The frequentist approach tends to be fairly easy to implement (which is perhaps why it is used more often), but as can be seen above, the underlying framework is very subtle and it can be quite easy to misinterpret the results (often by interpreting a probability as the probability of the truth of a proposition - e.g. the p-value falacy where the p-value is assumed to be the probability that H0 is true).

The Bayesian approach is generally the other way round - it is conceptually a lot more straightforward, and less prone to misinterpretation, but tends to be more difficult to perform as it often requires integral calculus, and we often need bespoke solutions for every problem. Some cases are reasonably easy to perform though.

A Bayesian can directly assign a probability to the truth of a proposition, e.g. the probability that the wheel is rigged. This is because for a Bayesian, a probability is a measure of (relative) plausibility, and is not necessarily defined to be a long run frequency. However, in this case, a better place to start is estimating how biased the wheel is, i.e. the probability of landing on black. Rather than have a single probability, a Bayesian would instead have a probability distribution representing their state of knowledge about the value of that probability (yes a probability of a probability). The reason for doing so is to correctly represent our uncertainty about the value of that probability.

An objectivist Bayesian might start from a prior belief that represented complete ignorance of the value of the probability, which I will call theta, in this case a uniform prior on the interval [0, 1]. For technical reasons, we would do this using a Beta(1,1) distribution

enter image description here

As observations arrive, we then update this prior using Bayes rule,

P(theta|obs) = p(obs|theta)p(theta)/p(obs)

For a roulette wheel spin, which can land on black or red, the likelihood p(obs|theta) is a Bernoulli distribution (or Binomial for multiple spins). The reason for using a Beta prior is that it is the conjugate prior for Bernoulli and Binomial likelihoods, which means that the posterior obtained by Bayes rule will also be a Beta distribution. If you work out the maths (which I can't do here as there is no LaTeX support), if you observed the wheel landing on black, then the posterior belief would be a beta(2,1) distribution (note you have ruled out landing on black is impossible, theta = 0):

enter image description here

if you observe the wheel landing on black a second time, you get a Beta(3,1) posterior:

enter image description here

and so on. Say we observe the wheel landing on black twice more, and red once, we would have a Beta(5,2) posterior

enter image description here

Note I have done this from memory, without checking, so I may have got something wrong - caveat lector!

A subjectivist Bayesian would have started with a more informative prior, encoding that it would be difficult to make a very biased wheel (and that a casino owner would have to be very stupid to try!), and use a Beta distribution that was peaked around theta = 0.5, say a Beta(2,2) distribution (which is not very informative):

enter image description here

Some people dislike the subjectivity of informative prior, however this prior is more in accord with rational expectations (knowing how roulette wheels operate) than the uninformatve flat prior. If we have the same set of observations, and push them through Bayes' rule, we end up with a Beta(6,3) posterior:

enter image description here

Note that the peak of this posterior is closer to 0.5 because of the prior knowledge imposed by the prior, which is peaked at 0.5. Note that you can see that the prior is effectively adding some "virtual" observations, one black and one red for the Beta(1,1) and two black and two red for the Beta(2,2) prior.

So, now to work out the probability that the table is rigged. Unfortunately the probability according to our posterior that the wheel is exactly fair, i.e. theta = 0.5, is zero. This is because theta is a continuous variable, so the probability of any specific value is zero. The solution is to define an interval around 0.5 that we would consider to be fair for practical purposes, say [0.45 - 0.55]. We can work out the probability of the table being fair by integrating our posterior over that interval

enter image description here

I worked this out numerically to be approximately p(fair) = 0.1317, implying that p(rigged) = 1 - p(fair) = 0.8683

Alternatively, you could use Bayes rule more directly, as in @causative's answer - but the difficulty there is that you need to know the degree to which the wheel is rigged to calculate p(obseration|rigged), or you could adopt a prior distribution for that as well and integrate it out of the calculation (marhginalisation). In this particular case, I think the approach I have given here is a bit easier to follow, but should give the same result.

Now this probability is the probability that the wheel is fair/rigged, assuming the statistical model is right, so while there is still an issue of what the threshold should be, reasoning in terms of probabilities is a lot more natural than reasoning with "significance" or "confidence" levels (which sound like statements about plausibility, but unfortunately aren't).

What I would do is adopt a minimum risk strategy, and work out the cost of accusing the casino of using a rigged wheel when it is actually fair, and the cost of playing at the casino assuming that the wheel is fair when it is actually rigged. I can then work out how much I would expect to loose by choosing to denounce the casino or not, which is based on the costs and the probabilities I have already calculated. So for Bayesian probability there is a rational consistent approach, which is know as Bayesian decision theory.

CONCLUSION

Frequentism is conceptually tricky and easy to misinterpret, but easy to implement. Unfortunately the threshold where we decide to denounce the casino is difficult to define and depends on prior probabilities, which frequentists often discourage us from using because they are viewed as subjective.

Bayesianism on the other hand is conceptually easier, but you often need to end up doing integral calculus, which is enough to put a lot of people off using them. Defining the threshold is easier because we can directly express the probability that the wheel is rigged, which a frequentist can't.

However, this doesn't mean that one framework is right and the other is wrong - both are useful and it is a good idea to have both sets of tools in your toolbox and know how to use them (and how not to misuse them).

9
  • 1
    Regarding your joke "The error the frequentist is making here is to use a significance level that is far too low because the probability that the sun has actually gone nova is vanishingly small.", did you make a typo here since H0 here is 'the detector is lying to us' which is equivalent to the extremely likely outcome that Sun is not exploding, so a competent frequentist ought to use an abnormally low significance level (compared to the usual 5% suggested by Fisher for most biology or social sciences related NHST cases) to let H0 not be rejected artificially easily like in the joke... Commented Jul 23 at 0:09
  • 2
    Importantly to address OP's sample size threshold question for a biased game as a frequentist, a power analysis is required in addition to the initial bias detection either from your NHST or a simpler confidence interval calculation based on normal approximation assuming at least 30 or 40 samples at hand. And are you coming back to cover the missing fully Bayesian posterior + credible interval + Bayes factor power analysis which are supposed to be more natural and superior to NHST significance level and confidence interval?... Commented Jul 23 at 1:09
  • @DoubleKnot, yes, I'll find a less ambiguous/easily understood wording. I read the question more as how many times does it need to land on black consecutively before we begin to have doubts about fairness, in which case, just waiting for a sufficiently small p-value would do. However the power analysis takes the other (equally valid) view of how many observations would we need before we have seen them. I have a blog post elsewhere about that, which I may link to. I am hoping to add the Bayesian side as well, time permitting.
    – user6527
    Commented Jul 23 at 9:15
  • 1
    "Unfortunately the threshold where we decide to denounce the casino is difficult to define and depends on prior probabilities, which frequentists often discourage us from using because they are viewed as subjective." Another way to look at it is that there exists a conjugate prior (or possibly a set of them) that is consistent with the manner the hypothesis test and significance threshold are chosen - the frequentist just doesn't know what it is, whereas the Bayesian makes that assumption explicit.
    – Ray
    Commented Jul 23 at 17:41
  • @Ray, yes, often a frequentist procedure (e.g. confidence interval) is numerically equivalent to some Bayesian procedure for some particular prior, but the problem is that can't be used as a justification/insight into the frequentist procedure because of the incompatible definition of a probability. However, I think it is better to stick to one framework or the other and be careful not to mix them as that is a source of many of the misinterpretations. Sometimes it is safe, but sometimes it isn't.
    – user6527
    Commented Jul 23 at 17:46
6

You start with "From a frequentist standpoint, each spin of a fair roulette wheel ..." (emphasis mine). The frequentist approach you describe is in response to this question, where "the wheel is fair" is a hard coded assumption. If you asked a frequentist "[when] should we reasonably start to suspect that the roulette wheel might be biased?", they would suggest a different approach.

Asking the Bayesian the real question you're interested in, and the frequentist a different one, doesn't seem like a good way to compare the two approaches.

3
  • I think that is a fair criticism of how the question is formulated :| But the final question seems to make clear that the OP wants to start from a frequentist point of view - so, really without making that assumption. It's just that this whole question imo can be answered simply by setting some significance level for a null hypothesis and doing the calculation...
    – mudskipper
    Commented Jul 21 at 17:00
  • Yeah, hypothesis testing is how one could approach it in Stats 101, but there is also some real detailed mathematical research on "when should I stop while doing a variable-length sequence of experiments". As your comment on the main question noted, the real answer is a bunch of technical details, but it's written looking for a fight between the two big schools.
    – JonathanZ
    Commented Jul 21 at 17:17
  • 1
    A quick bit of searching turns up the phrase "stopping rules" for the term of art in statistics. It also turns up mostly Bayesians! There's also discussion about using mixed models, combining Baysian and frequentist approaches. But it's all very practical and engineer-y, based on costs and risks, and doesn't seem to involve philosophy.
    – JonathanZ
    Commented Jul 21 at 17:28
6

You start with some estimate of how likely it is, before the spins, that the wheel is rigged to only roll black. Then you look at the chance the wheel would have produced the sequence of spins if it was fair. If the latter is less likely than the former, then you can be 50% confident that the wheel is rigged.

The prior estimate depends on where you find the wheel. If a friend wanted to show you something cool, then the chance the wheel is rigged is quite high. If the wheel was instead in some large casino, then the casino operator would not benefit from making it only roll black and would only stand to be sued and fined, so the chance it is rigged in that case is very low. The only way for it to be rigged in a large casino would be some elaborate sabotage by someone trying to discredit the casino.

So maybe 1 in 1000 if a friend wants to show you something cool (taking into account the technical difficulty of rigging it to only roll black), and 1 in 1 million for the casino. That would correspond to about 10 black rolls for the friend, or 20 for the casino.

Note: here's how you get this decision method from Bayes' theorem.

P(rigged | observation) = P(observation | rigged) P(rigged)/P(observation) = 1 * P(rigged)/(P(rigged) P(observation|rigged) + P(not rigged) P(observation|not rigged)) = P(rigged)/(P(rigged) * 1 + P(not rigged) P(observation|not rigged))

which is roughly equal to P(rigged)/(P(rigged) + P(observation|not rigged)) because P(not rigged) is roughly 1. So if P(observation|not rigged) is less than P(rigged), then P(rigged|observation) will be above 0.5.

0

0.1 cent.

There is no hard threshold that can be set, but depends on how much fair/unfair the real roulette is.

If it is only slightly unfair, then a large(r) number of samples would be required to infer unfairness with adequate significance; on the other hand if the unfairness is great, then relatively less number of samples are necessary to infer unfairness.

Usually, in hypothesis testing, some threshold(s) are preselected before testing the hypothesis (eg this is used at CERN experiments with the five sigma criterion). This means that different threshold(s) can make a different hypothesis seem plausible. In other words, accepted hypothesis is a function of the threshold(s).

Something similar has been asked on math.se How many rolls do I need to determine if my dice are fair?. Therein, similarly to this answer, number of rolls given depend on (preselected) significance level, test power and detected effect size. (One can use the tools used in that answer to get a numeric answer to your question)

Choosing significance level = .05, power = 0.80, degrees of freedom = num sides - 1, and effect size = 0.5, I find the following sample sizes to test the fairness of 6-sided, 10-sided, and 20-sided dice:

6-sided: 52 rolls

10-sided: 63 rolls

20-sided: 83 rolls


The real problem of statistics is how to know we indeed have a representative sample of some population. Statistics practice relies mostly on heuristics (eg random sampling) in order to address this problem.

1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .