If you are a true physicalist, then in practice, given the limits of time and process there is a largest number that will ever be used. That does not mean it is in principle some magical kind of limit, but numbers beyond that are simply irrelevant. But who are we, now, to decide what number that will be? Why not be halfway modest and act as if it is far beyond anything we can imagine? Why not plan for a long future?
You choose that direction, you can abandon infinity, but you have to allow for continual increase, anyway.
From a 'Nonstandard Analysis" point of view, the elements of sets like the Real numbers, or the integers with infinity are not real, but are axiomatic definitions masquerading as things. (Two is the property of having distinct things but as few as possible, etc.) Countable infinity, as the number with every number you have encountered as predecessors, but no immediate predecessor, is the shorthand for encoding continual increase. It is that biggest number ever used, forever getting away from us, slipping away into the future.
Any such axiomatic definition, expressed in a pattern that can be written down, is a recognition mechanism, that can be rearranged for use as a generating mechanism.
For instance, the native Intuitionist model (a la Brower) of a real number is "a freely flowing stream of bits." Every real number is process that will always hand you the next digit of precision. The number itself is treated like a point in space, but underlying it is really an ongoing approximation.
Given the notion that any rule can be looked at as a process, all other useful applications of infinity can be re-encoded in a similar form.
So it is perfectly reasonable to think of the numerical parts of mathematics as good thinking about measurements and approximations and their ultimate limitations even when you are 'taking limits as x goes to infinity', or dividing two things both 'going to zero'.
Things like infinite groups, etc. abstract that underlying mechanism away, assuming it can be captured faithfully in an intuition and ripped away from its more concrete forms. If you are not willing to make that leap, then you can stick to geometries and finite structures, and assume the nominally infinite ones do not have any applications that will interest you.
If you do make that leap, you have moved from computation to psychology. By making assumptions that human intuitions around things like infinity or continuity have an interest of their own, and that the fascination we feel for them has some basis, you can embark on a kind of deeply psychological art, either out of interest in the psychology, or attraction to the art.
Some of the products of that art turn out to have representations in reality, that make certain kinds of other things easier to imagine. Much like other kinds of stories help us get through life. But these stories are always 'Roman a Clef', we know where the characters come from. So the representations can be unwound back into finite terms and modeled in computation when they have genuine applications.
The question is why we can get from computation to art and back to computation easier by allowing ourselves a certain level of excess in the art than we can by sticking with reality. Basically, why is the human mathematical intuition a stronger tool than its motivation, if everything it models beyond its concrete applications is really not there?
It is the same question that makes language fascinating. If the universe is basically physical and evolution is what drives most of this, then why on earth would we evolve something so much more powerful that evolution itself, (solving the same kinds of problems hundreds of times faster) and then use that to create another kind of evolution altogether (competing ideologies and cultures)?
(Reality is enough. No one needs lies. But as Nietzsche points out, we have not yet begun to even estimate their power.)