Good question, and one involving the problem of the universals. Ultimately, knowledge, if such a thing can exist at all, must involve intentionality, that is it must be about something. That means that when we know something, it must involve apprehension of the thing known. When I say "Fido is a Dog", I possess a proposition that is about Fido and about "Dogginess" and about Dogginess being true of Fido. Indeed, for this proposition to be true, it must be about something that actually is the case; as the saying goes, "veritas est adaequatio intellectus et rei": truth is the conformity of the intellect with the thing.
Now, how can this kind of intentionality take place? You mentioned physicalism, which is interesting, because intentionality, as as species of telos (aboutness is a species of directedness), is banned from physicalism as an assumed tenet. Indeed, this is one of the decisive blows against physicalism. Others are the so-called problem of qualia and the inability to account for subjective experience (taken to absurd lengths like eliminativism, i.e., the denial of subjective experience), as well as the existence of abstract concepts (take Triangularity as opposed to concrete triangles in the world; you never encounter the first, only the second, but the second are made intelligible by means of the first which are abstracted from those concrete instances).
So, I would say, knowledge is immaterial.