The first part of this question is not entirely correct; but I can break it down from a physics perspective to show why.
What is true is that we can only 'remember' in one direction of time. This is because of the ONLY law of physics that seems to be dependent on time as an axis; the 2nd Law of Thermodynamics, commonly known as the Law of Entropy (In any closed system, the internal stated moves from more ordered to less ordered over time).
But wait you say; if I have MORE memories over time, then my brain is MORE ordered over time, is it not? True. But, the order in the universe is ever so slightly less because laying down those memories release a lot of heat into the environment (this is why wearing a beanie in a snowfield actually helps you stay warm). In other words, your brain is not a closed system; the universe is generating higher local order at the price of lower global order.
BUT, scientists believe we can predict not only the future, but those parts of the past we've never had the capacity to see. At least, on the universal scale. We have the Big Bang Theory for instance, a wealth of evidence for Evolution, and we are pretty sure we know how the earth will die as the sun expands, and how the universe will eventually end.
Why is this? Determinism.
We've seen enough of how the physical world works over time to determine certain laws like Special and General Relativity, Quantum Mechanics, and even Entropy. If these laws are 'true', then it means that the universe runs as an algorithm.
Algorithms are a special branch of mathematics that basically relate to complex sets of rules that if applied to a starting condition, will result in a known end condition. If the universe is algorithmic by nature, that means that if we perfectly know the state of the universe at any point in time, we can predict every other point (forward or backward) simply by applying the algorithm to the known data state.
This is how we know the Big Bang happened. It's also how we know that the universe is heading to a 'cold death' in the very distant future.
The problem here is that if we can perfectly 'predict' the future, it means that our future can also be perfectly predicted, because we are part of the universe. That means that everything we're about to do is already set. No free will.
So mathematicians like Roger Penrose postulate that the universe is non-algorithmic in nature, and that by expanding that definition, our free will can be restored by proving that the human mind and our consciousness operates in a non-algorithmic manner. This model means that the universe is mostly-deterministic; all the normal physical rules we've already discovered apply (because most of the universe operates algorithmically still) but it's also a blank canvas that we can change at will.
Let's assume for a moment that Penrose is right; it would seem to imply that the universe itself can still be predicted (especially into the past for things like the Big Bang) but that we can't be predicted, because we operate in a non-deterministic way. The problem with that theory is that if it's true, it means that the universe can no longer be predicted into the future because one day we might have the technology to change the universe on its own scale; at least change enough of it (say this galaxy) to preserve our lives in some manner.
Imagine a distant future where we've learned to mine black holes for exotic materials and energy. Imagine that we can alter the gravitational constant, bringing the universe either into a stable size (or even bringing it back in a bit first, to make interstellar travel easier). Imagine that we can put a whole bunch of mass together, split it apart into hydrogen, then compress it on solar scales to make our own stars and solar systems.
In such a future, we've now altered the universe beyond its deterministic constraints.
Even worse, if the human mind isn't deterministic, it means it's possible that the universe isn't either. That means everything we know about the beginning of the universe onwards might be wrong.
So; my take on all this is that we can either have the penny or the bun; either the universe (including us) is deterministic, meaning we can predict the future (although not remember it) but we lose all hope of free will being possible; OR the universe is NOT deterministic, meaning that our free will may not be an illusion, but the price is that we lose any certainty around being able to predict our past, let alone our future. Adding in even the smallest 'random' element, when you're going back 14.5 Billion years, may well be enough to make the Big Bang wrong, not to mention all of the physical laws we currently understand. (The probabilistic nature of Quantum Theory is not non-deterministic per se and has a special application whose description is out of scope of this answer)
Ultimately, until we have conclusive evidence one way or the other, it's up to us to choose what we believe; the math or our own experience. Trust me when I say that's not an easy choice.