Let us look at a model of what is happening.
Tim
-----xxxxxxx-------------------|
Husband (Tim)
??????------------|
Wife
----------------------|
One must explain the past of the husband in order to provide a coherent scenario.
Here are two scenarios in which the above occurs that do not contain the specified contradiction.
Suppose the existence of parallel universes which contain largely agreeing events. Then traveling to the future of another universe in which Tim decided not to travel in the future could yield this result. For the time traveling Tim originated elsewhere than the happily married Tim.
Suppose that 20 years in the future, someone created a copy of Tim from the past. The copy has all of Tim's properties from that time. Then according to the copied Tim, he traveled to the future since there were no intermediary experiences from then to now. But we still must think of copied Tim as Tim since he possesses all the properties that would identify him as Tim by assumption.