Everyone seems to have a notion of "physical space". How do we come to acquire it in our development, exactly? Every individual person has a visual percept, but how do we extrapolate from that to a public, objective physical space?
It's not necessary to have any intrinsic sense of space in order to perceive it, though you do need something that learns correlations. Space emerges from the correlational structure of perceptions. You can help things out a bit by preparing the substrate for noticing correlations in a space-filling way (e.g. self-organizing map), but it's really not necessary mathematically. For instance, a classic example of machine learning is taking images and producing what look like visual cell receptive fields (which are spatially extended). Practically every machine learning method that isn't simply categorical can do this.
On the other hand, parts of our brain are laid out to receive spatial information without any input. For example, visual cortex is laid out in a regular fashion to receive and process a 2D visual scene (from each eye), and correlations are sharpened pre-birth by spontaneous retinal activity.
The brain structure which handles most spatial recognition tasks is the hippocampus. Part of it is designed to handle spatial orientation as evidenced by grid cells (which only make sense for spatial environments), though it's not clear whether that's completely pre-developed or whether it could be a consequence of learning. However, the hippocampus is involved in all sorts of memory tasks including episodic non-spatial memory, so it's not clear what to take home from this.
Anyway, bottom line is--how humans actually do it is complicated. But if you want to know what is necessary, you only need to be able to observe correlations, and space falls out.
There is a thread running from Kant all the way to Daniel Dennett, and backed up by our current understanding of how decisions are made in the brain, that implies we do not so much experience space, as project it onto the world. From that point of view, space precedes experience and does not develop.
We do develop a sense of physics, but we are ready to move through space from birth. We do not have the ability to focus our eyes, or direct our movements, but we already know things are in places. We are more amazed to discover our hands are the things that produce our own sensations of touching our hands than we are to discover that Mommy's face moves around in a smooth way that we can predict, and that it gets closer or farther away instead of getting bigger and smaller.
Space is then an instinctual framework that humans share, that allows them to communicate about the world, and not an aspect of the world itself. Our overall difficulty dealing with discoveries about matter and space not being smooth, flat and infinitely divisible point up both how space might not be as real as we imagine, and how deep our intuitions about it run.
Every individual person has a visual percept, but how do we extrapolate from that to a public, objective physical space?
We don't extrapolate. We guess and then subject that guess to criticism. Why would you guess that objective physical space exists? Let's suppose you start without knowing that there is any difference between you and the outside world. You might try to do stuff and notice that the results are not always what you intend. You then guess that the parts of the world that didn't do what you want are not part of you.
There is no reason at all to think that people have the same ideas about physical space. Nor is there any reason to think that those ideas are created at one particular time in a person's life. People don't think about basic arithmetic in the same way, see