What's missing from the picture is entropy; the tendency toward greater entropy establishes an effective 'direction of time' that correlates with our perceived sense of time passing. We see eggs break rather than spontaneously forming from broken pieces. We can conceive of physical trajectories that would 'reform an egg' due to reversibility, but in practice, entropy means that these trajectories are too improbable to occur.
Partly this is a matter of how we perceive and categorize the world. We can't perceive all the microscopic details of most 'disordered outcomes', so we lump all those states together in our mind into a general sense: "the egg is broken". There are more total ways for an egg to be broken than for it to be whole, strongly favoring breakage once we overcome the energy barrier holding the egg in its metastable 'whole' state.
As for why perceptually we feel as if we're following the thread of causality along the path of greater entropy, well, that's an open question. Certainly we can talk about causality from within this framework that feels most natural to us, so it doesn't really make sense to say that it doesn't exist. It does certainly raise questions about whether this particular frame for causality is actually privileged, though, as it's not inconceivable that perception could operate in the other 'direction'.