The main objection is that it is (or most forms of it are) incoherent, inconclusive, or ignorant regarding the difference and/or relation between causal/physical aspects of sensing and justificational/normative aspects of knowing which is central to indirect realism
I think most philosophical objections are not against "indirect realism" (e.g. scientific realism) per se, but against a certain understanding of the justificational basis of our knowledge, i.e. representationalism in the form of foundationalist or coherentist epistemologies (these SEP entries include an overview of ((counter-)counter-)objections as well).
The locus classicus when it comes to objections against representationalism in extreme forms (foundationalism/coherentism, internalism/externalism) and a call for a "middle way" (DeVries/Triplett 2000, Introduction, xxxii-xlii) is Wilfried Sellars' Empiricism and the Philosophy of Mind (EPM).
As you write, representations of physical objects are often thought to be an observed fact, i.e. facts are "given" in representations by perception - this is what Sellars attacks under the name of "The Myth of the Given". Knowing by sensing is in itself a complicated matter. It is impossible to discuss the whole depth and spectrum of arguments developed in this long and dense essay, but I can quote how Sellars exposes the basic problem at the heart of representationalist theories:
Now if we bear in mind that the point of the epistemological category of the given is, presumably, to explicate the idea that empirical knowledge rests on a ‘foundation’ of non-inferential knowledge of matter of fact, we may well experience a feeling of surprise on noting that according to sense—datum theorists [a brand of representationalism], it is particulars that are sensed. For what is known, even in non—inferential knowledge, is facts rather than particulars, items of the form something’s being thus-and-
so or something’s standing in a certain relation to something else. It
would seem, then, that the sensing of sense contents cannot constitute
knowledge, inferential or non-inferential; and if so, we may well ask, what light does the concept of a sense datum throw on the ‘foundations of empirical knowledge"? The sense-datum theorist, it would seem, must choose between saying:
(a) It is particulars which are sensed. Sensing is not knowing. The existence of sense data does not logically imply the existence of
knowledge, or
(b) Sensing is a form of knowing. It is facts rather than particulars
which are sensed.
On alternative (a) the fact that a sense content was sensed would be
a non-epistemic fact about the sense content. Yet it would be hasty to
conclude that this alternative precludes any logical connection between the sensing of sense contents and the possession of non-inferential knowledge. For even if the sensing of sense contents did
not logically imply the existence of non-inferential knowledge, the
converse might well be true. Thus, the non-inferential knowledge of
particular matter of fact might logically imply the existence of sense
data (for example, seeing that a certain physical object is red might
logically imply sensing a red sense content) even though the sensing of
a red sense content were not itself a cognitive fact and did not imply
the possession of non-inferential knowledge.
On the second alternative, (b), the sensing of sense contents would
logically imply the existence of non-inferential knowledge for the
simple reason that it would be this knowledge. But, once again, it
would be facts rather than particulars which are sensed. (EPM, §3, bolded mine)
In other words: It is crucial how the story of how we become able to base veridical (and potentially fallible) observational knowledge of facts (cognitive, propositional, intentional, thus representations that are essentially normative and allow for justification within the "Space of Reasons") on the representations given by sense (nonpropositional, particular) is told if we are to maintain any form of empirical knowledge or realism.
The whole essay revolves around showing a) that the difference just described is a real and important one if we are to understand the epistemic authority of observational reports correctly and b) how different theories tell incoherent and fallacious stories that ignore or mingle the difference. And it gives a sketchy idea of how the alternative would have to look like.
For a good (the only?) clarifying commentary, see DeVries, W. A. & Triplett, T. (2000). Knowledge, Mind, and the Given: Reading Wilfrid Sellars's" Empiricism and the Philosophy of Mind," Including the Complete Text of Sellars's Essay. Hackett Publishing.