Suppose that you hold a true belief X, and suppose you could somehow be confident that if you held instead not-X, this would not have any other consequences on your life. Suppose a wizard is standing by, ready to cast a spell to change your belief in X to the false belief not-X. Would you be indifferent to whether the wizard casts this spell?
Not likely; you would prefer to continue to believe the truth. Morality, and especially harm to the individual, is in some sense about what the individual prefers. So your preference here gives some basis that it's a moral harm to believe not-X.
We could flip the scenario; suppose you initially believe the false not-X, and the wizard proposes to change your belief to the true X. Again you would refuse, this time because you wrongly believe not-X to be true, and do not want to abandon what you think to be the truth. You are still showing a preference for believing the truth - it is simply that you do not know what the truth is, in this case.
Moral harm is not just about preferences, but specifically about well-informed preferences. In the case when you initially believe X, you are better informed than when you initially believe not-X, and therefore your preference from the initial position of believing X is more morally relevant than your preference from the initial position of believing not-X. Your well-informed preference is to believe X.
This also allows us to weigh the harm caused by believing not-X in comparison to possible benefits that may arise from such a belief, such as a possible sense of community with others who hold the same false belief. The question is, which tradeoffs of veracity vs. community would you prefer, if you were well-informed about X and also well-informed about all the possible positive consequences of holding not-X?
We must also consider the possible negative consequences of holding not-X. Beliefs are hardly ever held in a total vacuum. False beliefs tend to lead you to actions that you wouldn't prefer if you knew better. For instance, belief in not-X could cause you to proselytize not-X to others. It could cause you to demand not-X be taught to children. This causes harm to people besides yourself. So for your sense of community with other not-X believers, you have potentially helped cause harm to many others.
Beyond proselytizing, there may also be other harmful effects of believing not-X. If not-X is in contradiction to certain findings of science, then you may become motivated to suppress those findings, and distrust science in general, and again spread these tendencies to others. Or perhaps critical thinking leads people to believe not-X, and therefore your belief in not-X leads you to disfavor critical thinking, and favor a more authoritarian culture.