But as I perceive, this does not apply to mathematics: we don't actually need axioms to be true, whatever you mean by "true". Axioms are just axioms. You either accept them and do the theory or propose other axioms and do a different theory.
A pure formalist can try to claim this but it fails because one needs to assume at least some basic properties of finite binary strings (or equivalent) to even be able to meaningfully talk about even plain FOL (first-order logic), not to say more complicated axiomatic systems. I say more in this post.
So everyone (except those who disavow logical reasoning completely) must believe in the truth of some basic axioms about finite binary strings, such as TC (theory of concatenation) (given here). This is the bare minimum. But this opens the door to natural questions of whether other sentences over TC (with higher quantifier complexity) are also meaningful and whether they too have boolean truth values. In some sense, the answer to this question determines whether you are a strict finitist or not. If you doubt that anything beyond purely finitary things, then you may be stuck with just TC. But if you think that quantifying over all finite binary strings preserves boolean statements, then you get TC plus induction, which is equivalent to PA (Peano Arithmetic).
That's not all. There are some fundamental facts of logic, such as the semantic-completeness of FOL for countable theories and the syntactic incompleteness of formal systems that can interpret TC. The first one simply cannot be proven without some tiny bit of second-order arithmetic beyond PA. The second one can be argued to require second-order arithmetic in order to have genuine meaning, even though PA itself can prove a suitably encoded version. I say more about this here. Note that the logician Peter Smith also points out that anyone who accepts PA ought to also accept ACA.
ACA is actually very low in the general 'hierarchy' of foundational systems for modern mathematics. Yet it turns out that every known application of mathematics in the real world can be expressed as some sentence that is provable in ACA, and conversely every theorem of ACA appears to be true when appropriately interpreted about the real world (at least at human scales), so we actually have good empirical evidence for this part of mathematics, and such evidence is actually necessary to provide meaning and purpose to mathematics beyond just formal symbol-pushing!
In other words, it is untenable to say that mathematics is just as arbitrary as say the rules of chess. And so we do have to face the question of whether the foundational system we choose is in fact meaningful, which is why the above considerations are all important.
What about higher mathematics that deals with very abstract objects? Well, logicians who have been concerned with meaningfulness of foundations have generally agreed that predicativist concepts can reach roughly ATR0 (which is slightly beyond ACA but nowhere close to Z2). Since adding closure principles can reach roughly Π[1,1]-CA0, some might argue that Π[1,1]-CA0 is true too, but that's shaky ground. What is clear is that Z2 is impredicative, so one can legitimately doubt its meaningfulness even if one believes it is consistent.
On the other hand, the average mathematicians do not care about predicativism and usually say that they use ZFC. What most of them don't know is that ordinary mathematics very naturally stays within a very weak fragment of ZFC known as bounded ZFC, where roughly speaking you can have inbuilt function-symbols for pairing, power-set, union, and inbuilt constant ω, and can construct any set of the form { E : x∈S ∧ ... ∧ y∈T ∧ Q } where E is a term and Q is a formula with only bounded quantifiers. A bounded quantifier in set theory is of the form "∀x∈A" or "∃x∈A" where A is some term.
If we grant full power-sets to be meaningful, we get past Z2 since the impredicative quantification over the power-set of ℕ would not be an issue. But even then there does not seem to be any non-circular way to justify much more than bounded ZFC. To reach full ZFC requires believing in the meaningfulness of impredicative quantification over the entire (complete) set-theoretic universe, and that impredicativity also implies that we cannot use the iterative conception to justify full ZFC.
If you ask ordinary mathematicians whether they believe their theorems are true or just the final state of a symbol-pushing game, they will most likely tell you they believe their theorems are in some sense truths that they discovered. But as explained above, even that evidence from actual mathematical work only reaches bounded ZFC. Of course, modern logicians often do use the full impredicative power of ZFC, especially when studying ZFC itself, but not all of them believe that there is a 'true set-theoretic universe' (whatever that might mean).
So, no, you really cannot escape the question of meaning and truth no matter how low or high you look in the 'hierarchy' of foundations of mathematics!