In the Tractatus, Wittgenstein rejected the Logicist program of Frege and Russell to define the concept of number based on logical notions only (included the extension of concepts, for Frege, and the theory of classes for Russell).
We can see :
4.1272 [...] one cannot say, for example, ‘There are objects’, as
one might say, ‘There are books’. And it is just as impossible to say, ‘There are 100 objects’, or, ‘There are ℵ0 objects’. And it is nonsensical to speak of the total number of objects.
for the rejection of Russell's Axiom of Infinity, necessary for the foundational project developed in the Principia.
And also 4.1273, for Wittgenstein's critique of Frege and Russell's definition of successor.
For Wittgenstein, numbers are not "logical objects":
4.128 Logical forms are without number. Hence there are no pre-eminent numbers in logic [...]
And 5.453 All numbers in logic stand in need of justification. Or rather, it must become evident that there are no numbers in logic.
Number is a sort of "primitive" concept:
6.021 A number is the exponent of an operation.
And :
6.031 The theory of classes is completely superfluous in mathematics.
In the Tractatus, mathematics (better : arithmetic) is essentially calculations, i.e. an activity based in signs manipulation (see 6.2 and on).
This point of view will be developed by later Wittgenstein in the theory of language games based on rules.