There is nothing wrong with it, materialism is a valid philosophical position. But it does not solve anything either. We are rather far away from explaining how concepts are represented physically. So you can simply state they must be so, and you can wave at brain configurations or something. But the problem remains to be solved.
We are even farther from explaining how social conventions are represented physically. You cannot wave at the English language and talk about brain states. But in some sense it still exists, spanning brains and living in the interactions between them. The language is not written text, or sound vibrations. It exists in an abstract way that allows for infinite configurations that are valid, and many more that are not. And its speakers believe that we can determined which are which. But none of those rules can claim to have a location. So what are they, physically?
Similarly, living things have a 'drive' that keeps them replicating. You can claim it is determined by the attempts of genes to create copies of themselves, but a gene is an abstract construct. There are really only chemical configurations there. The genome does not separate itself into genes. We do that. So how does this physical mechanism that knows nothing of genes managed this competition between genes? I clearly does. The abstractions we make about it do follow rules. But they don't "know" those rules. The rules, again, aren't anywhere, but they exist in some sense.
There is an essential difference between actual matter and its potential configurations, and there is a further difference between those configurations themselves and the regularities that can be discerned in which of those configurations we actually encounter. You can declare that information physical if you wish. But after you have made this pronouncement, what changes?
Does it remove any actual difficulties? Do abstract concepts that are too complex for us to compute anything about from our known physics become any more tractable than when we just assumed they weren't physical because they 'felt different'? No. We still need essentially different ways of talking about and handling conceptual patterns than the ones we have for controlling material.
In particular, the concept of infinity does not spontaneously dissolve just because we know that there are only finitely many things in our universe. And we do not perceive open-ended patterns as any less infinite. The potential is there, though there is no physics that applies to it. You can declare these infinities either nonexistent or physical, but either position nets you nothing of value. The former deprives mathematics of any explanation and the latter removes the ability to say that physical things have a location, and other handy things we generally assume.
This is a choice with no positive consequences. So by Occam's razor, it should therefore be a choice we leave unmade.