Leibniz defined perfection as "a simple, positive quality in the highest degree". A few lines down the paragraph of the OP quote Malcolm explains a key part of his objection by referencing Wittgenstein:
"I believe that Wittgenstein has shown in the Investigations that nothing is intrinsically simple, but that whatever has the status of a simple, an indefinable, in one system of concepts, may have the status of a complex thing, a definable thing, in another system of concepts."
On this view Malcolm wrote at length in his book on Wittgenstein, and Hauptli’s supplement to Philosophical Investigations quotes him as saying:
"Paragraphs 47 through 49 present a tour de force in philosophical criticism. What is attacked is the assumption of the Tractatus (and of much previous metaphysics) that the distinction between simple and complex has an absolute sense. In a variety of telling examples Wittgenstein shows very clearly that whether any particular thing is called a ‘simple’ thing or a ‘complex’ thing depends on accepted conventions, on decisions made for practical purposes, or on what comparisons are at issue".
There is some discussion of the simplicity objection in Adams's book on Leibniz, and Scott in Scotus, Malcolm, and Anselm notes the following:
"No doubt, Malcolm is taking demonstration in the sense of strict logical or mathematical proof leading to certitude. All one can get, so Scotus claims, is a 'persuasive' or probable proof. In connection with this, if Malcolm is dissatisfied with Leibniz' definition of perfection, he might weigh Scotus' remark that the main aspect of viewing the nature of God is infinity, not the summum bonum of Anselm".
On Leibniz's perfection probably the most comprehensive source is Howe's thesis Existence as a Perfection: A Reconsideration of the Ontological Argument, which is referenced and discussed in Smith's Doctrine of Existence as Perfection. More broadly, Leibniz's "modal" ontological argument and Malcolm's reconstruction of it are discussed in Lomasky's Leibniz and the Modal Argument for God's Existence. Look discusses the concept in Perfection, Power and the Passions in Spinoza and Leibniz, but not in relation to Malcolm's objections, he mentions that Schneewind in The Invention of Autonomy, gives a complementary discussion though.
