Is rarity, then, an illusion?
All statements related to probability/stochastic only make sense in the context of a mathematical theory with firm, consistent definitions. Tell me your definitions, and I'll tell you the answer.
In no case is this related to our perception of it - as long as you work in a given (and stated) mathematical framework, there is no "illusion", at least not in a more concrete sense that any high level maths tends to look like magic to mere mortals.
Even if you don't state a proper probability framework, each human has some intuitive understanding of it, which still is one of several frameworks - just unstated.
A secondary and possibly overly nitpicky aspect is that if you call something an illusion, you maybe mean that "it looks like it exists, but it does not really". Now we are in the realm of existence, which is a very deep area of philosophy for sure. In any case, to give a quick answer, I'd say "rarity" is a concept, or in the best case an attribute of something - but it is not something which "exists" in the first place at all; therefore it cannot be an illusion. (And also, if you talk about rarity of events, the same holds: an event is not an object or something that exists, so its rarity can hardly be an illusion at all).
So the only sense in which rarity can be an illusion is if we're talking fully intuitively, and then it simply means that our definitions do not match - person A talks a different "probability language" than person B, so the concept of rarity for person A might just not make sense to person B, but in truth it's a translation issue more than a philosophical one. And certainly not a deep philosophical, existential issue.
If so, why do we categorize some events as rare and others not?
Because the framework we chose to work in dictates that it be so (or not).
Note that the brain, for all of its wondrous capabilities is really not good with probabilities; especially with very small ones, and especially when combined with very large or very small "costs" of individual events. Intuitively, I would expect any intuitive statement about probability to be wrong in more cases than not. This is why industries like insurance or casinos thrive as well as they do.