We have to assume some finitary arithmetic, otherwise there's no way to even begin making sense of symbols, formulas, etc.
According to Hilbert, there is a privileged part of mathematics, contentual elementary number theory, which relies only on a “purely intuitive basis of concrete signs.” Whereas the operating with abstract concepts was considered “inadequate and uncertain,” there is a realm of
extra-logical discrete objects, which exist intuitively as immediate experience before all thought. If logical inference is to be certain, then these objects must be capable of being completely surveyed in all their parts, and their presentation, their difference, their succession (like the objects themselves) must exist for us immediately, intuitively, as something which cannot be reduced to something else.
These objects were, for Hilbert, signs. The domain of contentual number theory consists in the finitary numerals, i.e., sequences of strokes. These have no meaning, i.e., they do not stand for abstract objects, but they can be operated on (e.g., concatenated) and compared. Knowledge of their properties and relations is intuitive and unmediated by logical inference. Contentual number theory developed this way is secure, according to Hilbert: no contradictions can arise simply because there is no logical structure in the propositions of contentual number theory.
The intuitive-contentual operations with signs forms the basis of Hilbert's metamathematics. Just as contentual number theory operates with sequences of strokes, so metamathematics operates with sequences of symbols (formulas, proofs). Formulas and proofs can be syntactically manipulated, and the properties and relationships of formulas and proofs are similarly based in a logic-free intuitive capacity which guarantees certainty of knowledge about formulas and proofs arrived at by such syntactic operations. [...]
The question of what Hilbert thought the epistemological status of the objects of finitism was is equally difficult. In order to carry out the task of providing a secure foundation for infinitistic mathematics, access to finitary objects must be immediate and certain. Hilbert's philosophical background was broadly Kantian, as was Bernays's, who was closely affiliated with the neo-Kantian school of philosophy around Leonard Nelson in Göttingen. Hilbert's characterization of finitism often refers to Kantian intuition, and the objects of finitism as objects given intuitively. [...] Both Bernays and Hilbert justify finitary knowledge in broadly Kantian terms (without however going so far as to provide a transcendental deduction), characterizing finitary reasoning as the kind of reasoning that underlies all mathematical, and indeed, scientific, thinking, and without which such thought would be impossible.