I'm not sure what you mean by "can only go so far," but I'm happy to show how far one can infer from perceptions.
A perception is what you think you see, but what you really see are just sensations. If the physical cause of your perception is the thing you think you see, then there is a fact that corresponds to your perception; otherwise there isn't.
A perception itself involves inferences and is therefore not the ultimate origin of our inference. When you think you see a hard-cover book in front of you, you spontaneously assume there are many other qualities, such as hardness, persistence, crispness, impenetrability that are not in your immediate sensations. If you reach out your hand and feel nothing where the book is supposed to be, you will experience a shock (or will be surprised); this example shows that what you think you see is more than what you really see. What you really see is just a patch of colour; what you think you see is spontaneously inferred.
The ultimate origin of our inference is our sensation; most people, except perhaps painters, are not aware of the difference between what they think they see and what they really see. Sensations, except for very faint ones, are most indubitable, even when one is hallucinating; it is the inferences that are open to doubt. An untrained Child will pain a leaf solid green because he thinks he sees a leaf that is solid green; a painter will use white paint to show reflections although he knows the leaf is solid green.
“Science as the pursuit of truth is the equal, but not the superior, of art.” --Bertrand Russell. The Scientific Outlook.
It is no coincidence that this modern scientific age is heralded by the Renaissance, and the Renaissance began with the art. It is no coincidence that all great civilizations were preceded by great poets.
Another curious fact about perception is that everything you see over there in your perception is physically here because perceptual space is inferred from sensations, and sensations are limited to here and now (in the mind) - remember that the actual thing you see is a patch of colour in your own head.
In conclusion, it should be observed that the twofold space in which percepts are located is closely analogous to the two fold time of memories. In subjective time, memories are in the past; in objective time, they are now. Similarly, in subjective space my percept of a table is over there, but in physical space it is here.
Russell, Bertrand. "Space in Psychology." Human Knowledge Its Scope and Limits. New York: Simon and Schuster, 1948. 223. Print.
So can we infer from our perception to the thing that we think we perceive? If you sit in a security room, and the monitor shows a gunman walks into the building, can you infer that there really is a gunman at the door, or some prankster is playing an old movie on your monitor? Both are possible; both are open to doubt because there involves a leap of inference from what's on the screen to what really happens outside. We are not certain there really is a gunman, neither can we assert that there isn't such a gunman in virtual of that images on the monitor screen is not a real person.
Philosophical scrutiny only increases doubt. Yes or no, we are not certain either way. Down to the last analysis, we are still making do.