Why don't fair coin tosses “add up”? Or… is “gambler's fallacy” really valid?

Question

I have always been perplexed by a seeming paradox in probability that I'm sure has some simple, well-known explanation. We say that a "fair coin" or whatever has "no memory."

At each toss the odds are once again reset at 50:50. Hence the "gambler's fallacy." After 10 heads, the odds of another head are still said to be 50:50. The same after 20, 40, 80... heads.

Yet we also know that the series will converge upon an equilibrium of heads:tails. And indeed this is countable in fairly short order. The convergence appears pretty quickly.

How can both be true? Isn't there something in the physical series of tosses that "remembers"? Isn't there necessarily some slightly better chance of a tails after 10 heads?

How does logic resolve this absolute randomness in the particular events with a general law of convergence? I imagine this must be a well-known issue. I suppose it raises the larger issue of what sort of "causality" probability is.

Note that I do not know symbolic logic so, embarrassingly, formal demonstrations are beyond my ken.

Answer 1

Since you have asked for a non-formal answer, I shall try to oblige by not using any numbers or equations.

Fundamentally, your question is, how does it come about that individual events can be completely unpredictable but when you pile a lot of them together, either in a sequence or in a mass, the behaviour of the whole pile becomes, if not totally predictable, at least substantially predictable? The answer is something called the law of large numbers, and it is one of the most fundamental concepts in statistics.

As an illustration of it, imagine something called a Galton box: it is a triangular shaped box standing vertically, with its base on the ground and one vertex at the top. There is hole in the top to allow a ball to be dropped in. A series of pins or pegs are placed such that a ball falls either to the right or left in an unpredictable way until it reaches the bottom. As illustrated in this diagram, when lots of balls are dropped in, the result is a heap in the middle. We cannot predict where a single ball will fall, but put enough balls in and we can be increasingly sure that we'll get a bell-shaped curve, simply because it is very unlikely that a ball will consistently move left, or consistently move right. One way to think of it is to count the possible paths to a point on the bottom. There is only one possible path to get all the way to the right, or all the way to the left, but lots of paths will take a ball to the middle. This means we don't need to suppose that a ball is remembering the previous falls. Each ball is independent, and the resulting curve (a binomial distribution) emerges spontaneously from it. This is one of many examples of how apparently orderly behaviour can emerge even when there are lots of disorderly things going on at the micro level. Another is radioactive decay: we cannot predict when one atom will decay, but with a large mass of them we can predict very precisely what proportion of them will decay in a given time interval. Another example arises from the kinetic theory of heat: we cannot predict how individual molecules move around, but put enough of them together and we can say all kinds of useful things about their thermodynamic properties.

So the gambler's fallacy is a real fallacy, even though it is perennially tempting. My favourite way to test peoples' intuitions about it is to ask them this: suppose I decide to play the lottery every week and my preferred strategy for picking the numbers is to look up the numbers that won last week and choose those. You will find many people who think this is crazy because the chances of the same set of numbers winning two weeks running is tiny. But of course the probability of any set of numbers winning is all equal: it is not affected by the previous week's win.

Answer 2

If the probability of heads = p , then the probability of tails = 1-p . If it's a fair coin, then p = 1-p and the probability of either heads or tails is p = 1/2.

Now suppose the number of coin tosses is N, and let's say that N is getting pretty large. The expected value of the random variable that is the number heads out of the N tosses is going to be around the mean Np, which for an honest coin is N/2.

The variance of the random variable (the total number of heads out of N tosses) is Np(1-p) (which, for the honest coin, is N/4) which is the square of the standard deviation. This means if N is increased by a factor of 4, then the standard deviation only increases by a factor of 2.

So as the number of tosses increases, the deviation of the number of heads (which is sqrt(N)/2)) from the expected mean (which is N/2) does increase, but not as fast as the number of tosses increases. When you divide by N, the percentage of that expected deviation, inside the total number of tosses, gets smaller and gets closer to the expected 50%. This is because it's (sqrt(N)/2)/N = 1/(2 sqrt(N)) .

From a percentage POV, it looks like you're getting closer and closer to what is expected from an honest coin.

From a count POV, it doesn't look exactly the same. If you toss an honest coin 1,000,000 times, the number of heads will likely be some distance away from 500,000. But the percentage of the number of heads out of the total number of tosses will be very close to 50%. And it will get closer to 50% with more and more tosses, but the absolute distance away from the 50% mark will grow at a rate proportional to sqrt(N). But the number of tosses is growing at a rate of N.

Answer 3

The convergence appears pretty quickly.

This is your faulty assumption. It does apear pretty quickly. In most cases. But not at all every time.

There are in some sense two layers of likelyhood: In layer one, every single event has the very same probability as its precedessors. In layer two, the sequence of events as a whole has a probability to occur. And there are (finite!) sequences, although every single event is equally probable, that are as a whole unprobable. Only with infinite length every sequence is equally possible.

The problem is that you never know in "what" sequence you are, it may retrospectavely be the most unlikely sequence possible. That is why, looking to the future, only the probability of the single next event is what counts for the gambler, not the probability for the completed series.

You have to to see what the event (and object of probability) is. In the coin-example, the series so far is an event that has occured. So there was a probability for this series to occur before, but now as it has happened it is only a frequency of occurence left. The only probability is that of the next event, because it lies in the future, or of the upcoming series. But as soon as the next toss is made, there is only the probability of the now next event and the now following possible series.

Probability can only be expressed for future events!

As a sidenote the following "counter-example": Consider three doors, you choose one that has the "probability" of containing the prize of 1/3. Now one door is opened and you have the choice to change the chosen door. What are the chances? Well, you should definately change, because your door now has the "probability" of 1/3 as before, but the other has 2/3. Here you have to consider the whole series, there is no contradiction. That is because there is no probability anymore: The prize already is behind one door, the event has happened. That is the difference.

TL;DR: Edit and Conclusion

So the fallacy, as expressed by @wedstrom in his comment, is to think that nature will correct itself, will let it happen that the series in progress will converge quickly. But nature is not an actor that does anything. And in the present, there is only past (occured events, frequency) and future (upcoming events, probability). And if the probability is independent, this has to be taken literally as independence from anything that happened in the past, no matter how scarce the occurence of the resulting series will be.

Answer 4

Yet we also know that the series will converge upon an equilibrium of heads:tails.

I think this is your central problem. This is indeed the most probable result of a series of coin tosses, but probability doesn't apply to things that are already known to have happened.

Imagine this game:

A coin is tossed 100 times. Gamblers can bet on the total number of heads that will be thrown. They can do so at any time before or during the game.

Imagine that you're betting before the game starts. Your best bet is obviously 50 heads (50% of 100 future tosses).

Now imagine that you're betting after the coin was already tossed 10 times and came up heads all 10 times.

What is the best bet now? According to the gambler's fallacy, the coins should even out and so the best bet should still be 50. But in reality, the most likely outcome for the future tosses is still 50% heads, and we already have 10 heads, so the best bet is 55 (10 known heads + 50% of 90 future tosses).

Answer 5

If you use a fair coin, the average of heads thrown will converge to 50%. However, the number of heads won't converge to half the coins thrown.

While the percentage comes closer and closer to 50%, usually the number of coins will diverge more and more from exactly half. How can this be? Throw ten coins. You'll probably get 3 to 7 heads. 30% to 70%. Throw 1000 coins. You'll probably got 450 to 550 heads. 45% to 55%. Even though you are closer to 50%, you are actually further away (50 instead of 2) from having exactly half the coin throws being heads. No memory is needed. Your percentage comes closer to 50%, even though you actually deviate more.

Now throw 1000 coins, and then throw another 1000 coins. Say each time you have between 45% and 55% heads. But since there is no memory, there is a fifty percent chance that in the first 1000 throws you had less than 50%, and in the next 1000 throws you had more than 50%, or the other way round. In that case, you get a lot closer to 50%. For example, 45% + 55% means exactly 50%.

Answer 6

This is really math, not philosophy.

Assume that you've tossed the coin so far m times and gotten n heads. The fraction of heads so far is n / m.

Now you toss the coin one more time.

There is a 50% chance that the toss is tails and the fraction becomes n / (m + 1), and a 50% chance that the toss is heads and the fraction becomes (n + 1) / (m + 1).

By linearity of expectation, the expected fraction after the additional toss is then (n + 0.5) / (m + 1).

Now you can verify that if n / m = 0.5, then (n + 0.5) / (m + 1) = 0.5 as well — if we've had an even run so far, then the expected value after one more toss remains even.

If 0.5 < n / m, then 0.5 < (n + 0.5) / (m + 1) < n / m.

If n / m < 0.5, then n / m < (n + 0.5) / (m + 1) < 0.5.

In other words, if we've had an uneven run so far, the expected value after one more toss is slightly closer to even than it was before for no other reason than that the denominator of the fraction increases at a faster pace than the numerator does. You can start out getting 100 heads out of 100 tosses, but 100 independent tosses later you should expect to be at 150/200, which is closer to 50%. And 800 tosses after that you should expect to be at 550 / 1000. The excess is 50 in all three cases, but the percentage of excess got smaller.

Answer 7

Because "converge to an equilibrium" doesn't mean an exactly equal number of heads and tails, it means the proportion of heads to tails approaches equality (with probability 1: the meaning of which hides all the mathematical formalism to deal with the possibility of other results). In fact the probability of an exactly equal number of heads and tails after an even number of tosses tends towards zero with more tosses.

Ignore for a moment that there's an initial run of heads. Just start with the score "heads: 10, tails 0", and a fair coin. Then the score still "converges to an equilibrium", because the more coin tosses you make, the smaller is the proportional difference made by the unfair advantage of 10. You're happier to give someone a 10m headstart in a marathon than in a 100m sprint, and in effect tails is happy to give heads any amount of headstart in an "infinite race". As you approach infinity all fixed constants are small, the probability that tails has caught up with heads at least once along the way approaches 1, the probability that tails is ahead approaches 0.5, and that's all we mean by equilibrium.

The same goes for any initial sequence of coin tosses. Whether it's even or not, it gets buried by the unbounded sequence of coin tosses that comes afterwards. Consider, if you have the mathematics to do so, that the limit as x approaches infinity of (x+1)/x is 1. The numerator is given a "headstart" over the denominator, but it makes no difference to the limit.

Answer 8

You are comparing two different cases. One is "the probability of landing heads on the next flip" and the other is "sum of the number of heads." The latter is governed by the Central Limit Theorem, which explains why the sum converges so rapidly (in many cases). Summing acts very differently than simply asking "what's the next result," and its the summing that causes the convergence.

From the perspective of freeing ourselves from this "paradox" the key is that for every case where we have N tosses that landed heads, we also have a corresponding case where we have N tosses that landed tails. From the perspective of "sum of the number of heads," this matters. In the case where we discus "the coin has landed heads up 10 times in a row," it does not, because the fact that we have stated it has landed heads up 10 times precludes us from considering the case where it landed 10 times tails up. The 10 tails case doesn't have any effect on our discussion of the next coin flip because it simply didn't happen. We aren't interested in it.

It's a bit easier to visualize the non-paradox if, instead of counting the number of heads and tails, we assign heads and tails numeric values (such as +1 and -1) and take the average. Most humans find it easy to intuit that the average of a sample will approach the average of the random variable as N gets large.

This visualization can be done in many ways. One way is to look at all the different sequences of heads and tails that can occur. Clearly each sequence occurs with equal probability (with a fair coin). However, when you put these into "bins" based on how many heads you see, you find that there are many more sequences with an "average" number of heads than those which have extraordinary numbers of heads. This causes us to see average numbers more often than extraordinary numbers.

To give a concrete example, the strings of length 3: 0 heads = 1 string ({T, T, T}), 1 heads = 3 strings ({H, T, T}, {T, H, T}, {T, T, H}), 2 heads = 3 strings ({H, H, T}, {H, T, H}, {T, H, H}), 3 heads = 1 string ({H, H, H}). 8 total strings, each with a probability of occurring of 1/8. Thus, by addition, probability of 0 heads = 1/8, 1 heads = 3/8, 2 heads = 3/8, 3 heads = 1/8

Answer 9

You are right: After a series of 10, 20, 40, 80 heads the probability for another head is still 1/2. It is not slightly less or slightly bigger, it is constantly 1/2. Tosses have no memory.

To reconcile this result with the naive expectation one should take into account: The probability of a series of length 10 with 10 heads is (1/2) ** 10, which is about 1/1000. i.e. the probability to get such series when always making 10 tosses is 1/1000.

And the probability of 80 heads is accordingly (1/2) ** 80, which is about 10 ** (-24), a decimal with the ciffer 1 at position 24 after the decimal point.

Hence the contribution of such exceptional series to the limit on all series of equal length is exceptionally small.

Answer 10

To build on what celtschk pointed out (and possibly others, I haven't read all of them) with more examples, 'tend towards 50/50' is not something as in the next n throws will negate any off-set that is currently in place, it's rather, when n gets big enough any current off-set becomes insignificant.

I.e.

Let's assume you somehow manage to toss 100 coins and get 100 heads, but from now on, for arguments sake, let's say the coin tosses split exactly 50/50.

This means at 200 tosses, we'd have 150 heads and 50 tails, still biased to heads.

At 500 tosses, 300 heads 200 tails, still biased to heads, but less so.

At 10000 tosses, 5050 heads, 4950 tails, this is almost 50/50.

At 1000000 tosses 500050 heads 499950 tails, with that many tosses, this has effectively converged on 50/50.

This is the convergence you see, the error that is there initially just becomes insignificant the more tosses you add. There is no 'slightly higher chance' of tails.

Answer 11

You need to be careful to specify the question you are asking. Going forward, the coin has no memory and the chance of heads on any given toss is 1/2. Period. End. The convergence to the mean is because any excess that you have now will be washed out in much larger numbers. Say the first ten tosses come up heads. At this point, if I asked the most probable number of heads after 100 tosses the answer is 55. This is a little high. If I asked the most probable number of heads after a million tosses it is 500005, while before the first 10 flips it was 500000. As the standard deviation of the number of heads in a million flips is 500, an excess of 5 is no big deal. This is what the law of large numbers says. No matter what excess you have now, if you make enough more flips it will be very small compared to the standard deviation of the rest of the flips. Nothing makes it get closer to the mean, but the excess gets washed out when you consider the average.

Answer 12

Let's assume you have tossed ten heads, and you are about to do a million further tosses. What is the expectation of the difference between heads and tails? Well, it's ten, because you already have ten tosses, and the expectation for the future tosses is as many heads as tails.

Let's for the moment assume that in the next million tosses, you get exactly half a million heads, and half a million tails. This means the difference turns out to be exactly the expectation, as with the first ten heads, you have ten more heads than tails.

However, if you look at the percentage of heads, you'll find that since 500,010 of 1,000,010 tosses were heads, you've got about 50.00005% heads, and 49.9995% tails. So that's pretty close to equal.

But of course it is not exactly the same numbers of heads and tails. Isn't that a problem? Actually, rather the opposite: If in a million tosses, you get exactly half a million heads, and not a single one more or less, you should get suspicious. Because the probability of exactly half a million heads in a million independent tosses of a perfectly fair coin only is about 0.032%. Even worse, that probability even shrinks as the sequence gets longer, and in the limit of infinitely many tosses goes to zero.

The result from a random toss sequence of a fair coin will likely be somewhere around equally many heads and tails. Indeed, that range of head counts likely to be found even grows with more coin tosses. It's just that it grows slower than the number of tosses (that is, if you do twice the number of tosses, the range you'll likely find the number of heads is not twice at large; indeed it is only sqrt(2) times, or about 1.4 times at large), and therefore the range for the fraction of heads goes down.

Now that growing range of likely head counts means that with enough tosses, your initial ten heads will indeed be completely inside the range of likely counts, and that range will eventually be so large that the ten counts are negligible compared to the deviation caused by the random tosses.

Answer 13

Series will generally converge but there is always a small probability that a serie does not converge after a finite number of trials, so there is no contradiction. If you already had 100 tails the whole serie will converge more slowly. The interpretation of probabilities (degree of credence? Objective propensity? Frequency?) is an independent matter.

Answer 14

How can both be true? Isn't there something in the physical series of tosses that "remembers"? Isn't there necessarily some slightly better chance of a tails after 10 heads?

Note that I do not know symbolic logic so, embarrassingly, formal demonstrations are beyond my ken.

My way of seeing it is by counting the possible outcomes. Let say, you do 10 coin tosses. There are a lot of outcomes; Exactly 1024 of them (2 to the power of 10), of which only:

• one is made of only heads
• 10 are made of one tail and nine heads
• 45 are made of two tail and eight heads

...

• 120 of them contains two heads more than tails
• 210 of them contains one heads more than tails
• 252 are made of as many tails as heads
• 210 of them contains one tail more than heads
• 120 of them contains two tails more than heads

...

• 10 are made of one head and nine tails
• one is made of only tails

The general formula is obtained using binomial coefficients, but I skipped the formalism.

All in all, there is a higher probability to have approximately as many heads as tails because there are a lot of way to order an even mix of heads and tails while little way to order uneven mixes.

Note: this is related to the concept of entropy, as expected from randomness.

Answer 15

Not sure if this is the kind of answer you are looking for, but here is a non-mathematical, intuitive explanation.

The tossing of a coin, while random, is still composed of a chain of events that are themselves theoretically predictable to some degree - it's just that these events are very complex and the way they interact (and what they are) are not known.

For example, the outcome of a coin toss could depend on the following properties:

• The shape and crafting of the coin
• The material the coin is made from
• The way in which one moves their hand/fingers to toss the coin
• The physics of gravity, momentum, air resistance, and other environmental factors
• The material of the surface the coin lands on

And so on. If you were to know the exact way these properties interact and know the initial conditions for each of these properties, you may have a better sense of how the coin might land (impossible in practice).

In terms of your question, each coin toss is an independent event that cannot dictate the next event. This is because each of the initial starting conditions will be slightly different. But the shape of the object will have a big impact on the possible outcomes. Heads or tails is determined by the precise interaction of all the variables in the process. Because of the structure of the coin, only two outcomes are possible, and neither is really more likely than the other based on the interaction of all the variables which is driven by the shape of the object. What pushes it one way or another (heads or tails) has to do with how physics causes all the parts of the system to interact together.

This means that the contribution of all the other factors when it comes to nudging the coin one way or another is not sufficient enough to make either heads or tails more likely than the other. When it all adds up over thousands of samples you see that both are pretty much equally likely to happen and this is because of the interaction of all the variables involved in this physical system.

Answer 16

Yet we also know that the series will converge upon an equilibrium of heads:tails

We don't, actually.

At every flip the probability of the next flip being heads or tails is still 50:50. Can we not flip an infinite number of heads? We say we cannot, because the probability is small, that is, it is the limit as x->infinity on 1/2^x. Mathematically, we can say this limit converges to 0 (if we are in normal math land).

But let us now imagine a dart board that is the unit circle. We through a dart into the board, and it hits the board at a single, random, point. There are an infinite number of points, so the probability of hitting any individual point is 0. Yet we must hit the board somewhere! So, wherever we hit the board, at that point a probability 0 event happened. This would seem to show that not only does probability have no causal power, but even infinitely unlikely events can be forced to occur in finite time, with infinite possibilities.

So, if you flipped a coin an infinite number of times, it is true that we should expect an exact 1:1 equilibrium between heads and tails (for a fair coin), but we would also expect infinite runs of both heads and tails within the larger infinite set, and if you chose afterwards to just look at these infinite sets our expectation for all infinite sets would be violated. So we expect an equilibrium of heads and tails at infinity, but we also expect to be wrong an infinite number of times corresponding to an infinitely small portion of the infinite set of infinite sets.

Answer 17

The sequential tossing of a coin creates the impression of "history building." However, if we use the equivalent method, we will clearly see that no history (memory) is built. Let's take the case of tossing one coin 1000 times, the equivalent method would be to toss 1000 coins, one time. With this method, it is clear that no history is built, and if we examine the coins, we should find about 500 heads (or tails)!

Answer 18

Suppose I decided to draw a picture, and a drew a line like this: 'I', and then another one exactly like this after it, and then another, and so...

This would be a tedious picture, but this is like a memoryless drawing - with each line being placed as though it was the first line placed.

This is by analogy just like the throw of a fair coin or die, each throw being memoryless.

The question is are there other ways of throwing that take into account history? Sure, not with a dice or coin, but certainly with a virtual dice in a virtual world, and an avatar throwing it.

And this would be like a man doing a drawing knowing the line he placed before and knowing the line he places after, and the line he is drawing right now.

He has before him an aim, and behind him a history; and right now the moment drawn.

Answer 19

There are already quite a lot of good math-based answers here, but this is the SE for philosophy, so I'd like to offer a more philosophical one. I think the most interesting part of your question is:

Isn't there something in the physical series of tosses that "remembers"?

because the answer is a surprising "yes!" It just isn't the coin that's doing the remembering.

Suppose I flip a fair coin ten times and get the result 'TTHHHTHTTT'. Now suppose I flip the coin another ten times and get 'TTTTHHTHTH' instead. Nothing unusual so far.

But wait! Each of those two sequences is actually very unusual--in fact, the chances of either one are exactly the same as the chances of getting heads ten times in a row! An outcome like 'TTHHHTHTTT' only seems more "random" than ten heads in a row because your brain unconsciously throws out information about sequence. To our brains, the two outcomes 'TTHHHTHTTT' and 'TTTTHHTHTH' both just look like "messy jumbles of 'T's and 'H's," even though objectively speaking they are completely different.

So the reason you have an equal chance of flipping heads or tails even after flipping nine heads in a row, is simply that the two sequences 'HHHHHHHHT' and 'HHHHHHHHHH' are just as likely to occur as any other sequence of ten flips--that's the "fair coins have no memory" part. But what about the other part? Where does the law of large numbers come from, if all sequences of flips are equally likely?

I mentioned earlier that your brain unconsciously throws out information about sequence when looking at outcomes like 'TTHHHTHTTT' or 'TTTTHHTHTH', and this is why those two results look so similar. Well, the law of large numbers works because it does exactly the same thing! The law doesn't predict the exact sequence you'll get if you flip a coin a large number of times--rather, the law takes the total number of heads flipped, compares it to the total number of tails, then extrapolates that ratio for longer and longer sequences of flips. As far as the law of large numbers is concerned, the sequence 'TTHHHTHTTT' is exactly the same as the sequence 'TTTTHHTHTH'--or, for that matter, 'HHHHTTTTTT'--because they each have six 'T's and four 'H's, and that's all there is to it.

So in fact, the law of large numbers does imply "something that remembers"--otherwise there would be no way to keep track of the totals. The trick is that the "thing that remembers" is you! The law of large numbers relies on your memory for you to derive and make use of it. So in answer to the final part of your question, you might say that the "causality of probability" is just your expectations acting on past results: instead of saying the coin's fairness "causes" it to come up tails 50% of the time, you would say that your previous experience with fair coins causes you to expect the coin to come up heads or tails equally with each flip. (This is the general view taken by Bayseian probability, a fascinating branch of mathematics and one of many possible interpretations of probability.)

Answer 20

Most of the times the fallacy -and your problem- become true only when the events reduce the odds of getting the same values in the future, say:

My opaque jar has 100 balls. 50 of them are white, and 50 black. What's the odds of getting a black one when grabbing just one?

This event remembers the history and, if you pick them all, or you pick just one, the odds were the same: 50/50.

But your problem is the contrast between the uncertainity and the already-known events. All the times you should look at the definition of the problem. If the past is not a constraint (as it was in my example) then forget the damn past and move on:

I flip a coin. What's the odds of getting tails when I flip?

It says nothing regarding the past, because flipping an ideal coin has nothing to do with any physical property (a non-ideal coin, i.e. a real world one, perhaps gets their edges less sharpened when they hit the ground and the future result may vary...). Edit: Indeed, this wikipedia article related to entropy contains a graph with the coin flip distribution, and people knowing this would never commit this fallacy again since it is allowed to have coin flips where an ideal coin would have 1 in each flip, although that would only be a limit case.

Most fallacy-holders think the problem like this:

The coin has a quality of a true balance on a specific interval of experiments. If I flip it X times, X/2 of those times will have the desired result.

They take (or observe) the initial problem like this (without math jargon; otherwise they would not incur this fallacy):

• The prerrequisite is to chose either heads or tails.
• The experiment consists on flipping a coin and observing the result.
• It is common to get half times heads, and half times tails.

And convert them to this:

• The prerrequisite is to chose either heads or tails.
• The experiment consists on flipping a coin and observing the result.
• It is guaranteed to get half times heads, and half times tails.

(Most of the times they will know nothing about variance and SD, so there's no need to detail those concepts anymore).

Although the difference is subtle in language, is not subtle about what do you know about your system. You are changing the propositions and adding another constraint (yes: reducing the entropy).

So: Get back to the roots of your problem. Is your system evolving through experiment iterations? If so, you gain knowledge of the system, and you get closer to the overall initial information you know. When you reach that state, your entropy becomes 0 (exactly 0 shannons here): you know what the last ball is.

However, if your system does not evolve with iterations, the overall initial propositions still apply: Same experiment, same odds you already know (1 shannon over and over and over and over and damn over until our deaths and beyond or until the coin stops somehow being ideal).

Answer 21

Worth a mention is regression toward the mean, which is a real thing, although completely a-causal. It doesn't meant that if you've had a very unlikely outcome (8 heads out of 10 flips) that the probability of the next flip is biased against heads, but it does mean that it is >>50% probable that the average of any future sample-of-ten will be closer to 50% heads than your up-to-now outcome of 80%.

https://en.wikipedia.org/wiki/Regression_toward_the_mean

Answer 22

Anyone who sees 80 HEADS in a row and doesn't expect to see HEADS on the next toss is an idiot, and I would like to gamble with you.

The basic issue is that we don't actually know if the coin is fair or not, we can only make an assumption by assigning a prior probability of 50/50. We must then update our belief about fairness after each toss. The weight that you give to your initial assumption determines how little or how much you should change expectations given prior results.

Interestingly, in the specific case of a coin toss, we need not make any prior assumption about fairness to guess optimally.If the coin is fair, then it never matters if we guess heads or tails.

Therefore, optimal strategy dictates that we should always guess whichever result has been observed most frequently, even after a single toss. If the coin is fair, we lose nothing by doing so. But, if there is even minuscule advantage for one result over another, we are most likely to be right if we guess the most frequently occurring result.