When talking about things like integers, we can clearly ascribe a meaning to them. For example, 4 corresponds to "||||". The same is true of other finite objects.
This means, for example, that even if a statement isn't decidable in our given axiom system, its still clear that it is either true or false in an absolute sense. Take for instance the consistency of Peano Arithmetic. If Peano Arithmetic is inconsistent, we can write down a proof of 0=1. If not, then not.
This also means that axiom systems are either true or false. We can study PA plus the axiom that PA is inconsistent. This is a consistient system of reasoning, by Godel's second incompleteness theorem. Yet, it is incorrect. It asserts that there exists an object x which is a proof of 0=1 in PA, yet, when we put any particular object for x, the axiom is false.
My question is, what meaning should we ascribe to theories of infinite objects, such as set theory? For example, the axiom of choice is independent of ZF set theory, yet it seemingly very different the statement from the PA is consistient. Its not clear what it would mean for the axiom of choice to be true or false though. In fact, its not even clear what it would mean for the axiom of infinity to be true or false.
One way to approach the example of set theory would be to use a cumulative hierarchy, but that it itself dependent on ordinal numbers, an infinitary concept. If you had a theory of ordinal numbers, you could just use that theorem in conjunction with the cumulative hierarchy as your set theory.
EDIT: I guess I should say, is it possible to ascribe a meaning to infinite sets? If so, how would you do it. Can you give infinite concepts any meaning, or do they only have meaning relative to a formal system (which reduces them back to finitary objects, since the formal systems are finitary).