There are many lines of approach to this intriguing quotation. I'll mention just one, namely Freud's. (Because no-one accepts the whole of Freud, that doesn't mean he didn't have insights.)
Freud’s own memory was excellent. ‘‘I am not in general inclined to forget things,’’
he wrote in The Psychopathology of Everyday Life (SE 6:135). Yet Freud’s memory was
subject to the same failures and mix-ups that everyone experiences. For example, the
phenomenon he termed ‘‘cryptomnesia’’— what we might translate as ‘‘forgetting with
advantage.’’ With his customary candor, Freud recounted his own commissions of this
lapse. For example he described how he had been brought by his friend Fliess to realize
that he had completely blotted out the memory that Fliess had introduced him to the
theory of ‘‘original bisexuality,’’ a theory that he then later played back to Fliess as if Freud
had devised it himself. For most of us, lapses such as these function only as annoyances or
embarrassments. But Freud hypothesized that they could be made intelligible.
The first significant result of his inquiry into ordinary experiences of memory loss
and degradation was his essay on ‘‘The Psychical Mechanism of Forgetfulness’’ (1898),
which became the opening chapter of The Psychopathology of Everyday Life. Freud took
many of the analyses of mnemonic lapses in this study from his own experience, including
the classic case of his inability to recall the name of the painter of the Orvieto Last Judgment
frescoes. The result of his investigations strengthened Freud’s conviction that such
mental errors are always purposeful. So when Freud interpreted experiences of ‘‘forgetting,’’
he analyzed them not as simple memory drop-outs, but as blockages of recollection
determined by the psyche’s need to not remember something troubling. We could say
that these blockages then became memory-substitutes for unwanted recollections.
(Richard Terdiman, ''Memory in Freud', Histories, Theories, Debates. ed. S. Radstone & B. Schwarz, Fordham University, 2010.)
Maybe Freud generalised too much from a few examples but I find this passage illuminating even if we don't swallow the entire Fredian account of memory that
goes with it. The passage can be read as an explanation of, a rationale for, your quote : 'Memory is the thing you forget with.'