What you are relying on is called causal or reliabilist theory of knowledge that goes back to Ramsey and Goldman. But all of its theses can be and were rejected by various philosophers. "Direct observation" does not have to be restricted to sense perceptions, and causal effect does not have to be physical-on-physical. Plato had pre-birth mindsight, Aristotle had passive and active intellect for receiving universals and other idealities, Spinoza had intellectual intuition (“the human Mind has an adequate knowledge of God's eternal and infinite essence”), and Husserl had categorical intuition (and was a realist in his early period). Penelope Maddy, a prominent analytic philosopher of mathematics, is a mathematical realist, and so is her mentor Burgess. She follows Gödel, who explicitly adopted Husserlian theory of mathematical intuition, see How does a realist account for causation between universals and particulars?
Ideal perception operating beside sense perception is supported by modern cognitive science at least on the phenomenological level. As a result, Aristotelian realism about ideal objects and abstractions generally is on the rise since 1990s, and is advanced especially by the Australian school (Armstrong, Bigelow, etc.), see Franklin's review:
"The objects may be of any kind, physical, mental or abstract. The mathematical statement does not refer to any properties of the objects, but only to patterning of the parts in the complex of the four objects. If that seems to us less a solid truth about the real world than the causation of flu by viruses, that may be simply due to our blindness about relations, or tendency to regard them as somehow less real than things and properties. But relations (for example, relations of equality between parts of a structure) are as real as colours or causes".
So far nominalism and conceptualism did not offer an account of universals and our acquisition of them that is clearly superior to the Aristotelian one, although Wittgenstein's family resemblance is perhaps the most promising alternative, see Is it possible to use Wittgenstein's family resemblance approach to universals to separate high art from commercial art?
This should dispel the mysteriousness surrounding the nature and supposed causal inertness of the non-physical. As for "anything that has a causal effect on the physical world is itself physical", it sounds to me question begging against idealists, theists, dualists, etc., who would most certainly reject this premise. Other than Spinoza, I can not think of a monotheist of any kind who thought that God was a physical being, and even his phrasing ("Extension is an attribute of God, or God is an extended thing") is open to interpretation. For analogy to the God-world relation think of a "simulated reality". Within it we have "physical" existence and causation imposed by the program, but the programmers and the hardware exist "non-physically", and exercise quite a different type of causal influence on the simulacra. Leibniz and Berkeley present metaphysics were the ideal "simulates" the physical in a similar manner. But one need not believe in God or simulated universe, they serve as models, like a Euclidean models of non-Euclidean geometry, which show that even a materialist has to admit that causally effective non-physical entities are logically and even "metaphysically" possible. It can only be argued that they do not exist as a matter of fact.
Qualia and mental states in general are a different issue due to the role of the subject/object rift. There is an analog of the causal inertness concern for them articulated in Kim's causal exclusion argument, see Is there a causal influence of the mental on the physical? But even with them there is a Grand Canyon wide ambiguity on what counts (or eventually will count) as "physical". I think Nagel in What is it Like to be a Bat? gives the sharpest perspective on the issue:
"I have not defined the term 'physical'. Obviously it does not apply just to what can be described by the concepts of contemporary physics, since we expect further developments. Some may think there is nothing to prevent mental phenomena from eventually being recognized as physical in their own right. But whatever else may be said of the physical, it has to be objective. So if our idea of the physical ever expands to include mental phenomena, it will have to assign them an objective character — whether or not this is done by analyzing them in terms of other phenomena already regarded as physical. It seems to me more likely, however, that mental-physical relations will eventually be expressed in a theory whose fundamental terms cannot be placed clearly in either category".
See Are there philosophies that call for things which are not mind nor matter? and How do modern dualists explain the mind-body interaction? for a broader perspective.