I don't see what justifies naturalistic virtue ethics, or other forms of ethics that put forward virtues, for instance non-naturalistic command ethics.
To me, implementing an ethics that emphasizes virtues is inefficient and damaging:
When you are sick, you don't start from what does not exist (being in shape), but you start from what exists (being sick). Curing a patient consists in making them less sick (which finally results incrementally, consequentially, in making them in shape), not making them in shape (which I don't even know how it would look like concretelly).
If you implement the same logic of virtue ethics to sickness, it might result in people acting the targeted result, the state of being in shape (= having this or that virtue) (which does not exist, for it being targeted), which is inefficient and painful. An illustration of that is the episode of How I Met Your Mother where Barney does not want to accept he has a cold, and keep acting as if he were totally fine, while he is not.
If you implement virtue ethics, it might result in hypocrisy. Because it is inefficient for it being unrealistic, but because they are constrained to conform to it, people might play the role, without having really developed the targeted virtue. The person playing best the role of having this or that virtue will be the most rewarded.
For virtues in virtue ethics stemming not from a concrete reality, but from a general and abstract idea, it gives a lot of room to judge about whether an action or a person is characterized or not by this or that virtue. As a consequence, people might be confused before acting one way or another, while other people can freely argue that this or that past action, or their personality, is virtuous while it is not.
Implementing virtue ethics might restrain individual freedom, and the respect of individual integrity and dignity, and that resulting not in bringing greater welfare to the society, even going against it. Not all people are either brave, masculine (when being a man), feminine (when being a woman), generous, empathic, or this or that; but that doesn't mean this person should not be worth respecting, or allowed to develop and live their personality to their fullest. As Adam Smith, Bernard Mandeville and John Stuart Mill noticed, it is not only virtues, but often vices or peculiarity, that can bring the best human flourishing. An example is Steve Jobs who built one of the greatest company of all time, but who was by many testimonies, not really virtuous on many aspects.
To apply these different points on a concrete example. People criticizes the "dictarship of happiness", that makes people uncomfortable. I think we indeed find these drawbacks in our societies, precisely because happiness has been raised as a virtue, and not because it is seen as a positive outcome of a positive pursuit (that of happiness). Because happiness is seen as a virtue, and because virtue is intrinsically dependent on other people's judgement, there is a urgent social pressure to appear happy (most of the time, if not all), and not to be, or to become happy. Research has found repeatedly that intervening on the expression of emotions is maladaptive, detrimental, and inefficient. On the other hand, if happiness is seen from a consequentialist point of view, there will not be these drawbacks. People can freely find the ways that make them happy, and be unhappy when they need to. Human behavior is intrinsically related to physical health and mental health, therefore if one has to solve (behavioural) problems, same as when solving health issues, there is very little room for moral judgments. Instead, objective, detached methods, that work, should be applied.
What justifies "virtue ethics" or other forms of ethics emphasizing "virtues"?
NB: On the other hand, an ethics which is sober with regards to virtues (and deontology, the two often going hand in hand) (but holding one or two principles such as the harm principle and the principle of the absence of unnecessary pain as the goal of life*) might give rise to the appearance of many issues (since they are not forcefully being suppressed under the imposition of a limited code of conduct), but precisely these very concrete issues that individuals encounter during concrete situations and with their concrete personality or other traits can be tackled and solved, rather than forcefully hidden, resulting in an enhanced well-being and therefore enhanced morality. I think this is what we see in Western secular liberal democracies, with the debates and laws concerning LGBT rights, women's rights, abortion, euthanasia, animals rights, etc.
*the latter of which is somehow inscribed in the American 1776 Declaration ("that all men are created equal, that they are endowed by their Creator with certain unalienable Rights, that among these are Life, Liberty and the pursuit of Happiness") and the French 1789 Declaration ("afin que les réclamations des citoyens, fondées désormais sur des principes simples et incontestables, tournent toujours au maintien de la Constitution et au bonheur de tous.").