I am not in favor of vigilante violence against any political position, but, although your position is likely to be correct, your argument for it is not valid.
It is the slippery slope fallacy to say if you allow X at some level of Y, then X is going to be possible at a different, unrelated level of Y.
Violence that does not kill is not killing, so saying that to permit slapping your partner for embarrassing you in public facilitates murder is obviously not a well-founded argument. One might die from a slap, but that is not likely. I could make such a major miscalculation that I slap you hard enough you have a heart attack from shock and die. But making the call to slap you is still not my deciding to kill. To render those equally immoral is misplaced.
"Well, who would define that level of Y?" does not dispel the basic fallacy either. People do just define things. There can be measures in a moral theory that simply appeal to convention or human intuition. We can set a standard and allow for a reasonable level of misunderstanding and ambiguity so that we know using that standard is generally safe.
Consider the Aryans on which 'The Order' in the play God's Country are based. They propose a 'point system' that logically implies that to be a man in their society if there are any Black people left in this hemisphere, one must eventually kill one. And although its is a fictionalized amalgam, these rules did in fact actually exist for some Skinheads in the U.S. in the 1970's and 80's.
Using physical violence against people who have openly sworn to kill is not necessarily a problem. If you capture them via violent intervention and deliver them to the police, you may be stopping them from killing. And if they are just being rhetorical and do not intend to kill, you would be dissuading them from targeted terrorist manipulation. If that is part of your definition of what Nazis have to do to be Nazis, it is a safe one to use in this circumstance. No?
A label is not a criterion. So what someone is called is not part of a moral deduction. People lie, especially about their enemies, and especially when using hyperbolic labels. But what they are doing or have done, are saying or have said can be used as valid criteria to decide how to treat them.
These people probably can't really choose a definition of Nazi that works and applies to Charlottesville. But if they could, they may well be able to support their position in some way. Your dismissal of all possible arguments they might make is highly likely to be right, but is still logically premature.