One of the aspects of modern physics that makes time difficult to explain is that all of our current theories predict that things should be reversible, except for the second law of thermodynamics. This is broadly known as CPT-symmetry in physics, and results in a philosophical problem known as Loschmidt's paradox.
The problem is that this law is simply empirical, not a theoretical result of other theories. The detailed mathematics that holds it up, generally known as 'the fluctuation theorem' would work just as well if we consistently saw time going backward, and it does allow for time to reverse in regions of space with extremely high entropy. It just predicts this should be very uncommon.
Boltzmann's original way of looking at this was to predict reversals in time, and in fact to presume that at some point in time entropy would be maximized somewhere in space and time would in fact reverse there and this reversal would spread out to include the whole timeline. This has problems of its own, in particular, it favors the notion that the Universe never began, and we came into being as "Boltzmann Brains".
But ignoring that, since we can move forward in time and laterally in space, if the law of entropy reversed, we should be able to move to that place, at some point in our own timeline after the event of reversal there, and ride that timeline backward, then come back to our own region of space. We would not necessarily end up in a part of space-time from which we could get to our own past. But we would in fact have gone backward in time.
(Of course, in such a chaotic place, we would probably fall apart, and if we suddenly reversed all of our energy-exchange processes, we would probably not be able to function or control anything -- the physics for which all of our matter is evolved would no longer work the same way. So having an object, much less a person, survive this process is likely to be impossible.)