It is surely not clear that your initial statement is correct. We can generally divide phenomena into three categories: the effective (information), the reflective (internal constructions) and the affective (emotional responses). These distinctions have a lot of different names, but they seem to be a stable set of distinctions made in language (Germanic modal verbs), religion (Gnostic theories of the trinity, the three Pillars of the tree of life) and a variety of forms of pre-scientific magic (astrology and alchemy in their late Western forms).
In an absolute logical sense, only the first two are 'real', one can be either a realist or an idealist. But the third category always gets thrown in there, and is generally the main ground of the topic of ethics, the most specifically human part of philosophy.
From one perspective, the reason that the third category is generally not seen as part of either of the other two is that it is partially interior reprocessing and partially mental experience of the body (according to the James-Lange theory of emotions and its more modern improvements). It feeds back into itself so quickly that it becomes, to a certain degree, completely mixed, neither interior nor exterior and so independent of both of the preceding forms.
If that is true, then to the degree that they are partially observations of your body's internal condition, its contents are not really learned from other people. What we tend to learn instead is how to name and channel them, not how to feel them. According to studies of adoption, (Not the greatest reference, but -- http://adoptionvoicesmagazine.com/my-second-mama/do-adoptees-have-more-problems/#.VIXXBTHF-So) if your own emotional landscape is significantly different from your parents' you tend to grow up undercontrolled, rather than underemotional. So your theory seems backward.
To my mind, this is still not a good basis for judging someone else's level of emotional expression - they are either physiological or environmental, not moral. Lower levels of physiological feedback are not a crime, they do not make you less human. Nor are very effective strategies for predicting and restraining responses a commentary on your humanity.
However, emotional empathy is a basic form of communication. According to attachment theory, the ability for one's own emotions to inspire empathy in others is their basic purpose. To the degree others do not feel your emotions, they may find you to not be communicating clearly. Communication is a basic way of being human, so they have some justification in the feeling you are withholding your humanity from them.
It is also true that overrestrained emotion is often an overadaptation to strong emotion enforced by a family with a history of trouble keeping normal levels of emotion from growing excessive and violent. So a lot of people are frightened by a lack of emotional display, as it can be a predictor that someone will overreact in some later situation, perhaps violently, or perhaps simply at the worst possible time. If what your emotional presentation causes in most other people is fear, then you may be unconsciously manipulating them, and they are then justified in looking at you as something of a criminal, who gets his way by using their negative emotions against them, and so 'less human' than someone with a more average presentation.
So basically, they could be wrong, or you could be wrong, or there could be nothing wrong. But this is not as simple as you would like it to be, primarily because you have an interpretation of the source of emotion that most of us do not find compelling.