I second SAHornickel's reply above. And, here are some examples from Part V of Wittgenstein's Remarks on the Foundations of Mathematics that he was specifically thinking about:
§1. It is of course clear that the mathematician, in so far as he really is 'playing a game' does not infer. For here 'playing' must mean: acting in accordance with certain rules. And it would already be something outside the mere game for him to infer that he could act in this way according to the general rule.
§2. Does a calculating machine calculate?
Imagine that a calculating machine had come into existence by accident; now someone accidentally presses its knobs (or an animal walks over it) and it calculates the product 25 × 20.
I want to say: it is essential to mathematics that its signs are also employed in mufti.
It is the use outside mathematics, and so the meaning of the signs, that makes the sign-game into mathematics.
Just as it is not logical inference either, for me to make a change from one formation to another (say from one arrangement of chairs to another) if these arrangements have not a linguistic function apart from this transformation.
§3. But is it not true that someone with no idea of the meaning of Russell's symbols could work over Russell's proofs? And so could in an
important sense test whether they were right or wrong?
A human calculating machine might be trained so that when the rules of inference were shewn it and perhaps exemplified, it read through the proofs of a mathematical system (say that of Russell), and nodded its head after every correctly drawn conclusion, but shook its head at a mistake and stopped calculating. One could imagine this creature as otherwise perfectly imbecile.
We call a proof something that can be worked over, but can also be copied.
§4.If mathematics is a game, then playing some game is doing
mathematics, and in that case why isn't dancing mathematics too?
Imagine that calculating machines occurred in nature, but that people
could not pierce their cases. And now suppose that these people use
these appliances, say as we use calculation, though of that they know
nothing. Thus e.g. they make predictions with the aid of calculating
machines, but for them manipulating these queer objects is
experimenting.
These people lack concepts which we have; but what takes their place?
Think of the mechanism whose movement we saw as a geometrical
(kinematic) proof clearly it would not normally be said of someone
turning the wheel that he was proving something. Isn't it the same
with someone who makes and changes arrangements of signs as a game;
even when what he produces could be seen as a proof?
To say mathematics is a game is supposed to mean: in proving, we need
never appeal to the meaning of the signs, that is to their
extra-mathematical application. But then what does appealing to this
mean at all? How can such an appeal be of any avail?
Does it mean passing out of mathematics and returning to it again, or
does it mean passing from one method of mathematical inference to
another?
What does it mean to obtain a new concept of the surface of a sphere?
How is it then a concept of the surface of a sphere? Only in so far as
it can be applied to real spheres.
How far does one need to have a concept of 'proposition', in order to
understand Russellian mathematical logic?
and, this too:
§48. Does a line compel me to trace it?--No; but if I have decided to
use it as a model in this way, then it compels me.--No; then I compel
myself to use it in this way. I as it were cleave to it.--But here it
is surely important that I can form the decision with the (general)
interpretation so to speak once for all, and can hold by it, and do
not interpret afresh at every step.48. Does a line compel me to trace
it?--No; but if I have decided to use it as a model in this way, then
it compels me.--No; then I compel myself to use it in this way. I as
it were cleave to it.--But here it is surely important that I can form
the decision with the (general) interpretation so to speak once for
all, and can hold by it, and do not interpret afresh at every step.
The line, it might be said, intimates to me how I am to go. But that is of course only a picture. And if I judge that it intimates
this or that to me as it were irresponsibly, then I would not say that
I was following it as a rule.
"The line intimates to me how I am to go": that is merely a paraphrase for:--it is my last court of appeal for how I am to go.
The point is that justifications come to an end somewhere; there is no continually higher court of appeals. This sort of notion is connected here with rule-following and what the grounds are for justifications. These sections below are from On Certainty:
§471. It is so difficult to find the beginning. Or, better: it is
difficult to begin at the beginning. And not try to go further back.
§472. When a child learns language it learns at the same time what is to be investigated and what not. When it learns that there is a
cupboard in the room, it isn't taught to doubt whether what it sees
later on is still a cupboard or only a kind of stage set.
§473. Just as in writing we learn a particular basic form of letters and then vary it later, so we learn first the stability of things as
the norm, which is then subject to alterations.
§474. This games proves its worth. That may be the cause of its being played, but it is not the ground.
§475. I want to regard man here as an animal; as a primitive being to which one grants instinct but not ratiocination. As a creature in a
primitive state. Any logic good enough for a primitive means of
communication needs no apology from us. Language did not emerge from
some kind of ratiocination [Raisonnement].
That is to say, the grounds for appealing to a higher court or a sounder foundation come to an end: how do we know how to follow a rule? --It is not a question of "knowing." In §287 of OC, W. states that because the "squirrel does not infer by induction that it is going to need stores next winter as well...no more do we need a law of induction to justify our actions or our predictions." We cannot and need not appeal to justifications ad infinitum.