If that were the basis for allowing it, your logic would hold water. But it is not the basis for allowing it. And in fact (as my terrible, rambling answer to the given related question tries hard to clarify) "the obligation not to interfere in consensual acts where no one else is involved" is a meaningless argument, without a lot of extra analysis of the meaning of 'involved'.
Given a very literal sense of 'involves', consensual sex between two people who are married to other people involves no one else. If their spouses never found out and they could guarantee no harm would befall them due to this liason, then, this logic would say they should go ahead. And that is nonsense.
You need some standard for 'involves' that is broad enough to rule out this case. If you are married, your spouse must somehow be automatically involved in decisions you make to give consent to sex.
If you are swingers, any children should also be automatically 'involved' in such a decision. Perhaps not directly as participants in the negotiation, but via your obligations to them.
You and your spouse might adopt an open marriage. But you should not do so in a way that endangers your children's reliance upon you as a couple. For example, you should not mix partners in a way that might allow a paternity suit by a temporary partner to take children away from you as a couple who genuinely consider themselves yours, and depend upon that connection. Doing so disrespects your role as parents.
This is not even necessarily about sex, just about family identity. The same should be said of shady adoptions. If you adopt a child in a way that allows the biological parents to claim him back, this is exposing him and any real or potential siblings to unnecessary potential trauma. Planning for permanence in the parental bond is part of the deal. This obligation is only compromised when doing so is clearly in the best interest of the child, not when the parents or others choose to risk it for selfish reasons of their own.
Another part of the deal that traditionally comes with the family is that children should have a space safe from the pressures of sexuality and its related power issues. That place, in Western tradition, is presumed to be the extended family. So unless you make extraordinary plans to create that space elsewhere, you should forbid incest within the circle that is meant to constitute that safe space. Having, for instance, one set of siblings pair off romantically, breaks down the whole presumption that family provides a safe place for sexual innocence. So these people are involved, when you make decisions about marriages.
Gay marriage does not challenge the traditional safety that parents are obligated to extend to their children in the way incest does. So these are not parallel arguments at all. They involve very different standards as to who is 'involved'.