Imagine that the Universe had a temporal beginning but no temporal end. At the beginning the Universe has a finite size, and as time passes its size increases exponentially. And the number of observers is proportional to the size of the Universe, so as time passes the number of observers also increases exponentially. If I'm a random observer, then I should expect to be temporally located infinitely far away from the beginning. And then if, instead of travelling forward in time like everyone does, I could travel backward in time, I would never reach the beginning of the Universe in any finite amount of time.
Would it be logically possible that the Universe has a beginning in time but we're temporally located infinitely far away from the beginning?
By infinite I'm not merely saying that there is an infinite amount of instants that separates us from the beginning, by viewing time as continuous or dense. I'm saying that there is an infinite amount of seconds that separates us from the beginning. Time has to be viewed as the Natural numbers, not the Real numbers.
And I use the word infinite in the mathematical sense, so it doesn't just mean "a huge number" like a googol.
I don't really care much about relativity, the Big Bang, etc. I'm not asking whether it is actually the case that our Universe is like that, I'm just asking if this is philosophically possible or if it breaks the rules of logic.