The answer to this will depend on the metaphysical commitments you have. It is hard to see how a physicalist, or signatory to any form of metaphysical monism, i.e. to the belief that reality at root consists of one kind of stuff, can find room for numbers to be revealed to us through experience without being experienced themselves.
If everything is reducible to physics, then so is the number we count to when we see two apples; if this is physical in nature, then we experience this just as we experience their redness or acidic taste. Someone with this belief might further want to say that numbers are not things that exist apart from the physical entities that they bear on, rather that they supervene on top of these 'bona fide' physical things. The twoness of two apples is just a relation that obtains between these physical things, rather than something separately existing.
Just some thoughts to get started with.