This is an intriguing question. Context is importance; scruples often ought not to prevent us from reading the private letters of public figures. But I will keep to the kind of domestic situation you evidently have in mind.
Someone might say that there cannot be a rights-based objection since the dead, qua non-existent, have no rights. Is this so, however? Is it really the case that the dead cannot be rights-holders, morally? All else equal, don't the dead have the right to dignified interment, the right against unjustified disturbance after interment, the right to bodily integrity, and (given that a will comes into effect only when the testator is dead) the right to transfer property?
It's true that reading the letters might alter your moral view of the person - might lead you to think better or worse of them - but then isn't that (or might it not be) a justifiable alteration of perspective? I don't think the dead have a moral right to have their 'memory' protected come what may.
Suppose the letters are of a particularly intimate kind (sexual or erotic, it may be) such that they were intended for one other person's eyes only. In reading them, are you showing disrespect for the dead? Do the dead have a right to privacy? I'm not sure on what basis such a posthumous right would rest. I violate your privacy if I gain unwanted access - access unwanted by you - to personal data such as a letter you have written. But the dead are not capable of wanting to control access to such data and so in reading the letters you are not interfering by gaining unwanted access to data the dead wish to protect.
Others may think differently. Here I have simply offered a response derived from considerations that seem to me relevant about rights, rights-holders, and privacy.
B. Rössler, The Value of Privacy, Cambridge: Polity, 2005: 8.
Fred O. Smith Jr, 'The Constitution after Death', Columbia Law Review , OCTOBER 2020, Vol. 120, No. 6 (OCTOBER 2020), pp. 1471-1548: 1475.