Yes, the flow of time is exactly of the nature of a qualia, but I dispute the idea that physics has no answer about how or why we experience it. The thing that one must accept first is the physicist's positivism: once you answer every question about the measurable attributes of the brain and behavioral states, you have given the answer all questions about the qualia, even though the map between the qualia categories and the measurable attributes might be complex.
The theory of mind that physicists usually accept without reservations is the computational theory of mind, which was proposed by Alan Turing in the 1940s. The computational theory of mind begins with the observation that a computer can simulate the behavior of any physical system, including a human being, and then makes the positivist claim (this shouldn't even have to be said) that if you have something behaving indistinguishably from a conscious person, it is a conscious person. Turing codified the positivism in the Turing test--- if the communication with the computational entity is indistinguishable from the communication with a person, one identifies the computational entity as a conscious thing, like a person. This is common sense, and it is not something I consider worthy of reasoned debate.
The computational theory of mind identifies the flow of time with the direction of the computation, the relation between inputs and outputs. In any physical computer, the computation must dump heat into the environment in order to keep its information from randomizing, so for a regular computation in time, this is the same as the entropy arrow of time. The identification of the entropic and conscious arrow of time follows.
The "feeling" of time passing is a high level property of the brain's computation, and it is a property of the software, or the mind, or the soul, I use all these terms interchangably because they are equivalent in the computational view. The software is abstract data, and the manipulations of this data can be described using larger qualia, which are just shorthands for classes of data stored in the software.
The embedding of the software into the physics matches the conscious time sense with the physical time sense. The main complicated stuff is in the software, not in the laws of physics. That the flow of time is not physics is manifested by its dependence on the mental state of the observer--- having a traumatic experience requiring immediate action, taking hallucinogenic drugs, falling ill, each of these can change the perception of time in drastic ways.
The feeling of time flow is so universal, and the changes in this flow in unusual states is so jarring to the individual, that many people look to physics for confirmation of the feeling that time is something fluid and unreal. They find false confirmation of this in relativity (which has nothing to do with the essentially nonrelativistic events in the brain's computation) and string theory, and this makes these topics more popular than type II superconductors, although these are equally interesting and both are about equally relevant to consciousness.
Anyway, the person who removed time flow from physics explicitly and put it into the mind is Ludwig Boltzmann, and once he did so, he was able to ask questions about Boltzmann brains. The concept of a Boltzmann brain (a brain emerging fully formed from a thermal state by a fluctuation, and feeling continuity with previous experience vastly removed in physical time) requires a separation of the sense of cause and effect or of time from the laws of physics.
This is standard physics philosophy, it reappears in the many-worlds interpretation, which is controversial for reasons that have nothing to do with its computational theory of mind.