So there is some combination of three different things going on here. This is ad hominem in that the label is personal and irrelevant, it is Buverism in that it diagnoses the argument, and it is ad baculum control in invoking the personal consequences of arguing as if they were an argument themselves.
Even thus rephrased, the statement is obviously still ad-hominem. It calls out an attribute of the speaker as an insult instead of addressing the argument. Whether you presume or deduce racism is present is irrelevant. One can be racist and correct.
But this is a subcategory of ad-hominem so common it deserves a name. So C.S. Lewis gave it one. Its special feature is that uses that identity as a diagnosis of a flaw in the argument that may not exist. Lewis' original example is "You only say that because you are a man". (He invokes the mythical Mr. Bulver's mother as a stereotype of women who invoke the notion of men as lacking empathy on almost any occasion to diagnose his father as not being able to understand the issue at hand because of his gender.)
Lewis realized that Bulverism can exist without an ad-hominem launching pad, relying on an appeal to authority, the "fallacy-naming fallacy" or some other basis for the spurious diagnosis. So it is a separate fallacy. But it always has to exist in a combined form, and this particular combination is really common.
There is also a threat implicit in some categorizations. The risk of being identified as an atheist or a heretic used to shut up people about issues touching theology. Reagan's press secretary notoriously tried to shut down questions about the spread of AIDS by implying anyone who cared about the issue was gay... This is the personal form of 'ad baculum' -- 'arguing with a stick', which goes beyond a mere 'ad hominem' argument. It ends a potentially valid line of argument by raising a personal real or psychological threat against the person making it or those around them. Nowadays, being marked as racist or sexist can be used ad baculum, and that may be what is intended here.
There is a sense in which political moves like this belong more to rhetoric than logic, and do not pretend to be arguments. But by my standards this is not just rhetoric, because it controls the acceptable responses and shapes the logic of the argument.