Number do exist as long as the countable objects exist, irrespective of the observer, but if there is no observer what loses its meaning are not the numbers but the "meaning" (understanding of reality) concept instead!
That is to say, numbers as adjectives can exist without observer (like in "two apples"), but as an imaginary abstraction no as the abstraction itself is a work of the observer's mind.
Also counting needs some order whose definition very much depends on the observer (which apple in the basket to be assigned 1, which 2, and etc.), but that a basket has 9 apples need no observer and will keep its meaning even in the absent of the observers.
However, if you ask about zero, infinity, cardinals larger than infinity, negative numbers, non-rational numbers, then one can continue discussing things further. All I said above was about the rational numbers, although I believe everything that we discover has or potentially can have a place in the real out universe.
And as a last point, the physical rules of nature exist irrespective of being witnessed by an observer or not. Such rules have always numbers with themselves (like Pi, e, and maybe many more) even though there be no observer writing down the discovered rules on a piece of paper. That Einstein said Nature is numerically integrating somewhat clarifies this idea better.