This question is similar, but not identical, to one I posted to the mathematics SE some time ago. I was originally unsure of where to post it. I believe this question is sufficiently different to warrant not migrating the other one.
There is a famous quote in von Neumann's The Computer and the Brain (published posthumously), that goes:
When we talk mathematics, we may be discussing a secondary language, built on the primary language truly used by the central nervous system.
Also
Thus logics and mathematics in the central nervous system, when viewed as languages, must structurally be essentially different from those languages to which our common experience refers.
I've read the relevant passages a few times, and far as I can see he is trying to convey a statement along the lines of: "The mathematical or logical langauge of the nervous system is not the same as that we use when we do/talk about mathematics."
From a modern perspective, we may say that the nervous system is a biological neural network, with (as von Neumann himself suggested in this book) low precision but high reliability. And then, presumably, this neural net is Turing-complete, and the actual way in which computations are performed is not important - they are still computations, and they could just as well have been run on some sort of a cellular automaton.
The people over at the mathematics SE seem to think there is no formal mathematical substance here, and that the issue is purely philosophical.
So my question is, are there philosophical positions that make this feeling of von Neumann more concrete? What, if anything, does this add/take way tofrom a theory of mind? I am thinking particularly here about the computational theory of mind.
Was von Neumann onto something fundamental here about the nature of mathematics as a language?