The last century or two of mathematics made a lot of progress by making clearer what we mean when we say anything that contains "infinite" or "infinity". One can in fact always rephrase such statements in other terms. (This typically involves saying certain things can be done with arbitrarily large finite quantities, or even arbitrarily small ones; it's complicated, but well-worn.) In doing so, we explain what we mean in a finite amount of information.
In general, descriptions, definitions etc. don't inherit the properties of what they talk about, because descriptions are a very specific kind of thing. To take a less mysterious example, descriptions of blue aren't blue; and descriptions of descriptions of blue aren't descriptions of blue either.
Even though there are infinitely many integers, an integer can be specified with finitely many bits of information: first specify its number of binary digits (so you specify how many digits that number has, until we get down to 1), then what they are. In general a real number takes infinitely many digits to specify (or is unspecifiable, if you prefer to define "specifiable" as "specifiable in a finite number of digits"), but you can show two real numbers differ by citing just finitely many of them, until a difference comes up.