# Is there a term for the fact that it may need more information to describe a probability distribution than conveyed by the event itself?

For example, X is a random integer from 1 to 16. Now I get a piece of information: X is 3, 5, 9, or 14. This has 2 bits of information for the knowledge about X. But if the list of options is random enough, I'd have to list all four integers to describe this knowledge, which is much more than 2 bits. It may need more information, if the probability distribution is not uniform, and I attach a probability to each of the options.

Is there a term for this phenomenon?

In mathematics context, this is basically just saying, for a random variable X, there are infinite possibilities of Yi that I(X;Yi) is a given number, so if Y represents the pair <i,Yi> where i is also a random variable independent of everything else, I(X;Y) = I(X;Yi), but H(Y) could be made very high. Or for simplicity, let's ignore the first part, and say for random variables X and Y, H(Y) could be much higher than I(X;Y). This is a quite dull question in mathematics, because they are clearly different as we use different symbols to refer to them, and it is usually unambiguous which one we are talking about. But in philosophy, we could easily say "some knowledge about X", that could refer to either of them, that is the real information inside X, and the information inside X and of the approach to find X together. So I think something is required to address this problem.

In the example, the approach could be to categorize the values into 4 arbitrary lists of 4 elements each, and find which list it belongs. It has more information if we consider the approach not predecided, but chosen in a way using an extra random variable.

It's especially important in philosophy context because if we are discussing about the concept of knowledge, the approach is usually an open set, and there isn't a default one if nothing is specified.

• Are you talking about needing a prior distribution to get the (posterior) distribution after conditioning the prior on the "event itself"? How is the amount of space needed to record the options and their probabilities relevant here? Commented Oct 16, 2023 at 13:48
• What do you mean when you say "X is 3, 5, 9, or 14" has two bits of information? Commented Oct 16, 2023 at 14:42
• @DavidGudeman It has reduced the 16 options to only 4 options. Commented Oct 17, 2023 at 1:33
• This just sounds like Shannon information aka surprisal or information content of some event x consisting of several outcomes within a probability distribution, defined as -log P(x). In your second case with a greater range of the distribution the probability of the same said event would most likely decrease, ergo the surprisal now becomes larger… Commented Oct 17, 2023 at 7:05
• I’m voting to close this question because it's not about philosophy but about math. Commented Oct 17, 2023 at 10:23