As we have seen of late, and as Jonathan Haidt so adeptly pointed out in his The Coddling of the American Mind, the concept of harm has essentially been reconstituted to include things like having your feelings hurt, being offended, having your sensibilities violated, having your lived experience invalidated, being caused to feel unsafe, etc.
And while neither expressly defend the notion that what constitutes "harm" for purposes of limiting freedom of speech, should be determined exclusively by the experience of the individual upon whom the harm it is supposedly inflicted, the philosophical doctrines of transcendental idealism and of phenomenology, by validating only the objects of our experience, and invalidating non-inferential knowledge of real world objects/states of affairs, would seem to lend support to such a position.
Kantian Transcendental idealism (Critique of Pure Reason (1781)) posits that:
…the conscious subject cognizes the objects of experience not as they
are in themselves, but only the way they appear to us under the
conditions of our sensibility. Thus Kant's doctrine restricts the
scope of our cognition to appearances given to our sensibility
[phenomena] and denies that we can possess cognition of things as they
are in themselves, i.e. things as they are independently of how we
experience them through our cognitive faculties [the inaccessible
“nomena]. (See Wiki).
And phenomenology, which came on the continent to be prized as the proper foundation of all philosophy, is essentially the study of Kantian “phenomena” :
…appearances of things, or things as they appear in our experience, or
the ways we experience things, thus the meanings things have in our
experience. Phenomenology studies conscious experience as experienced
from the subjective or first person point of view. (See SEP).
Having found their way into 21st Century popular culture (and in the process given birth to a host of related doctrines: from pragmatism to identitarian epistemology to the modern popular notion of "lived experience"), these ideas have essentially rendered any adherence to positivism, objectivity, realism, even rationality, naïve, and allowed popular culture to slowly slip into epistemic nihilism, the position that there is no truth at all.
All of this has given rise, as I have here noted before, [the cultural ethos of] “post-truth,” the OED’s word of the year in 2016, the use of which had risen over 2000%; defined as:
“…relating to or denoting circumstances in which objective facts are
less influential in shaping public opinion than are appeals to emotion
and personal belief.”
From this history/narrative it is not surprising to find the popularly held sentiment which, as the OP suggests, by oversimplifying the complexity of the issue concludes that “what constitutes ‘harm’ for [purposes of limiting freedom of speech] should be answered exclusively by the individual upon whom it is supposedly inflicted.” Given where we currently [to some extent rightly] are epistemically, who else would be qualified to do so?