I hear the notion of a point being the most primitive notion in geometry. But to talk about a point, one needs to think of a space of some sort. Only then, the point can be understood as a position within it. This also mirrors set theory nicely since space can be thought of as a set and a set is the primitive notion in set theory. But are there other problems that arise by taking space as the primitive notion in geometry?
There are many geometries. Most of us are familiar with the Euclidean, having Descartes' coordinate grid superimposed on it. This grid naturally leads us to think in terms of points. Set theorists are apt to follow that convention. But the real world is closer on a large scale to the Minkowski spacetime geometry of Relativity theory, modified on a minute scale by quantum theory - we doubt whether the idea of a zero-dimensional point can have any physical meaning.
In projective geometry, both point and line are equally primitive. A point may be defined as the set of lines which meet in it, known as a "pencil" of lines. Dually a line may be defined as the set of points which lie on it, known as a "range" of points. Ideas such as plane and space arise by allowing a point which is not on a given line, or dually a line which is not in a given point, and similar axioms. These dualities arise as a theorem of projective geometry, so when you have allowed or proved one statement you have also allowed or proved the dual statement. One can therefore derive a set-theoretic model of projective geometry using lines as a primitive instead of points.
The Euclidean and non-Euclidean geometries (elliptic, hyperbolic, affine) may be derived from projective geometry in various ways, but they all break projective duality, i.e. duality is not a theorem any more but just arises in certain circumstances.
So the answer depends very much on the particular geometry or class of geometries you are thinking of.