If one is involved in a discussion such as the logical problem of evil or the logical possibility of philosophical zombies, if an opponent makes a formal logical error this needs to be corrected. The content of the argument is about the correctness of the logic. The audience expects both sides to correct the other's logical errors.
If one is faced with an informal fallacy such as the irrelevance fallacy of ad hominem one has to pay especial attention to the audience. Irrelevance and ambiguity (see Copi's classification) side-track the argument. Your goal is to get the argument back on track, not correct the opponent.
Bo Bennett warns if you do decide to accuse an opponent of an informal fallacy:
...what you certainly should be prepared for, is your opponents pointing out your fallacies....
Pointing out informal logical fallacies may only further distract focus from what is important in the argument.
Rhetorical discourse assumes there is a wider audience than the immediate opponent in an argument. One must keep this audience in mind when responding to any informal fallacies.
Bennett, B. "Being a Smart-Ass" Retrieve on May 31 from Logically Fallacious at https://www.logicallyfallacious.com/tools/lp/Bo/LogicalFallacies/207/Being-a-Smart-Ass
Copi, I. M. Introduction to Logic. Macmillan. 1982.