One position in the philosophy of maths is Structuralism (as has been pointed out in the comments). According to Structuralism, mathematics is not about numbers, sets, functions, etc., but about structures. It identifies common mathematical objects with places and relations in structures.
Some time ago, people thought that numbers are really sets. For example, 0 is the null set, 1 is the set of the null set, 2 is the set of the set of the null set, and so on.
But this way of identifying numbers with sets is just one way to do it. Here is another. 0 is the null set, 1 is the powerset of 0, 2 is the powerset of 1, and so on. (which gives us the following sequence: {}; {{}}; {{}, {{}}};...
(This is a very short and probably bad summary of Benacerrafs 'What Numbers Could Not Be' (1965))
This insight motivates the Structuralist's position, that any sequence of stuff will do.
But what are these structures? According to some, they are abstract objects in their own right. Here we have the problem of supplying a satisfying epistemology of abstracta. According to others, they are concrete. Here we have the problem that mathematics is thought to be non-contingent, while what is concrete, and how much concrete stuff there is, is contingent. And according to a third position, they are merely logically possible, which again dodges some problems and runs into others (do we know that there are possible structures? etc)
Literature:
The relevant SEP link is here
Resnik (1981) - Mathematics as a Science of Patterns
Shapiro (1997) - Philosophy of Mathematics - Structure and Ontology
Hellman (1989) - Mathematics Without Numbers - Towards a Modal-Structural Interpretation