To some extent, the underlying problem here (and something that Cohen tacitly discusses in the linked paper) is one of Demarcation. Where does the mathematics of Set Theory end and the logic of mathematical truth begin?
I really liked Cohen's description of the Lowenheim-Skolem theorem on p.1085 as
the ﬁrst nontrivial result of logic.
To summarize, so as to keep this answer relatively self-enclosed, the L-S theorem is a result in classical first-order logic that shows that wherever we have a transfinite model of some theory, we can show that there is a countable one, and indeed one of any arbitrary cardinality. This is a logical result, not a set-theoretic one, in the sense that it concerns a property about the abstract satisfiability of theories rather than something particularly semantic. In fact, the traditional understanding of this result seems at odds in many ways with what set theory tries to say about mathematical resources beyond the finite ordinals - a seeming called the Skolem Paradox (SEP Link).
Cohen knows about and understands this theorem incredibly well - it is importantly involved in the technique of forcing over Set theory models to get "new" models through a countable treatment adding new "generic" sets. This treatment is in some sense also independent of its set-theoretic interpretation, since what really matters for the purposes of using forcing isn't that the models are set-theoretic, but more that they follow a Boolean-valued model structure, which guarantees that there is some partial ordering on how we add new forcing conditions respecting classical negation closure and exclusion; set-theory is simply the most standard interpretation and representation of boolean valued models.
The question of whether Cohen's technique challenges the coherence of a full formalization of Logic is thus a very nuanced one. Using Forcing, Cohen shows us that given some accepted groundwork in the axioms of Set theory, it is possible to construct a diversity of models, some of which may adequately reflect hypothetical "axioms" or desirable properties that mathematicians may want to further dwell on, and others which may demonstrate properties not immediately reflected in how mathematicians go about practicing their craft. There may not necessarily be a sense of a right and wrong way to formalize our axioms, in that these various models might all be independently mathematically interesting, and all of them can be looked at and theorized about within the scope of mathematical logic as it currently stands.
But on the other hand, the logic that he uses to demonstrate this technique doesn't itself appear to be under threat by that observation, for the simple reason that the modality of Forcing only branches out in the realm of the transfinite, thanks to the Lowenheim-Skolem theorem. Nothing in forcing extensions will tell you anything unusual about whether
5+7=12 - they might intuitively semantically differ on what set, exactly, the sum of ordinal 5 and ordinal 7 is (unless we also have the axiom that we have a countable standard model), but thanks to the way forcing preserves satisfaction of the ZF axioms from model to model, truth of the relativised versions of statements about the finite ranks of the set theoretic hierarchy is preserved between forcing extensions of ZF models.
What this seems to turn on then is what kind of notion of Consequence is at stake when we look at the question of what makes something Logic versus Mathematics. An intuitive sort of "Truth across all models" conception would say that there is something seriously up with what Cohen is doing to logic here, because he's essentially creating "new" logical models that he ought, surely, to have recognised prior to using the L-S theorem as a logical result. But that doesn't seem to be what Cohen thinks he's doing - let's take a section from p1089:
Nevertheless, the feeling of elation was that I had eliminated many wrong possibilities by totally deserting the proof-theoretic approach. I was back in mathematics... Thus I assumed immediately that I had a standard model of set theory, which although 'obvious' cannot be proved in ZF.
Cohen's results although logical are properly mathematical consequences, rather than logic; Cohen via Godel had managed to get a certain amount of the way in using tactics in pure logic, but needed to step outside for additional power to get his interesting results in his classification of mathematical structures. The notion of mathematical truth seems to be a very particular species of truth once you get beyond a certain accepted kernel of logic (this would be in line with Hilbert's ideas about the role of consistency proofs of theories defined over finite arithmetic).
If this is right, then the idea that we have "settled" on first order classical predicate logic can be considered consistent with Cohen's reflections on the vast plurality to be found in set theory, because what we agree on and what logic in abstraction from strictly mathematical content can decide for us is that stable finite core that all of the possible models accept. That would be my way of lending some credence to the thesis that logic has in some sense been properly formalized.
But maybe that's not all there is to say; maybe we might want further to consider that we can add more logical structuring and higher cardinals to talk about degrees of truth according to how typical some propositions are across the various forcing extensions that we might come across. If you want to check out some of this more esoteric analysis, you could do worse than checking out some of the ideas in Woodin's Ω-logic (Bagaria et al have an advanced Primer for further reading).