I think the answer to the question hinges on what ethical framework we are working from. Put it another way, I think there's a hidden assumption that makes your though experiment appear to work.
If you read, for instance, Locke or Hobbes, or most contemporary political philosophies, they distinguish between the way membership in a state works and the way voluntary association works. Your thought experiment seems to assume states are like voluntary associations. But if we add this assumption then it's unsurprising that we see the forced involvement as illegitimate.
To give a similar analogy, in a family, there are adults and children. Is it fair that the adults work, but the kids don't have to?
It's nearly impossible to answer that until we start adding assumptions. To many of us, it would be obvious kids are not required to work in the same way. We may disagree, however, as to why since we may think kids are luxury goods created by parents or that kids are owed certain duties by those who choose to have to them. Or we may just see this as the model of care that nurtures children into the sort of people who can support themselves.
For the political case, much hinges on what we think a political society is and whether we think it is natural or not. There's two streams of philosophical thought on this. (Restricting ourselves to the Western tradition) There is one starting from Plato is that society is to some degree an unnatural extension of the rational self who can work in tandem (here I am thinking less of the Republic and more of say Phaedo, Meno, and Apology Socrates). A second stream building on Aristotle looks at society as the basic unity of human existence and sees the family and the individual as parts of this greater whole (Politics 1).
A second and related question is whether we view society first and foremost as a contract entered into by rational agents or some thing else (for instance, a church, or a nation).
In some interesting ways, the U.S.A. was originally an experiment in the idea of minimal government that enabled maximal free association and independent choices. This was partially inspired by Locke's idea of tolerance (See "Letter Concerning Toleration") and augmented by the experience of many of the groups that came to those shores as previously persecuted groups.
Since then the guiding philosophy for many later constitutions has been the idea of a people with a deeper bond than self-selection or the idea that the cosmopolitan ideals are sufficiently worthy of selection so as to force these onto others, but beneath this is the basic idea that humans don't exist primarily as individuals but rather as people in communities and societies, i.e. units in a greater whole rather than complete entities that then contract together.