In Platonism and Neoplatonism, dialectic assumes an ontological and metaphysical role in that it becomes the process whereby the intellect passes from sensibles to intelligibles, raising from idea to idea until it can finally grasp the Supreme Idea.
"Plato is not the man to dabble in abstract theories and principles;
his truth-loving mind has recognized and represented the truth of the
world in which he lived, the truth of the one spirit that lived in him
as in Greece itself. No man can overleap his time, the spirit of his
time is his spirit also; but the point at issue is, to recognize that
spirit by its content" -Hegel, Lectures On The History of Philosophy
Hegel's model of dialectic is almost always given as a simple threefold process, thesis meets antithesis with tension between resolved by synthesis; however Hegel himself did not do so. He used as the steps: abstract, negative, concrete. That arguably points more clearly to why the first step initiates a dynamic, and is insufficient alone. Trial, error, and experience, gets to the concrete. To get to the absolute, the discourse subject must have passed through negation, and been mediated to it. For Hegel it is not by synthesis, but by overcoming (aufhebung) of the negative, that the dialectic subject processes - the 'negative of the negative': Something is only what it is in its relation to another, but by the negation of the negation this something incorporates the other into itself. He claimed the dialectic is proceeding in that way, to make implicit contradictions within things, explicit.
Kant was hostile to the idea of the dialectic as route to transcendental truths, or the Absolute:
"Aristotle can be regarded as the father of logic. But his logic is
too scholastic, full of subtleties, and fundamentally has not been of
much value to the human understanding. It is a dialectic and an
organon for the art of disputation." -from his Lecture On Logic
He held to universal transcendental truths, as a Transcendental Idealist, and saw the dialectic as misleading us into metaphysics, and away from the grounding of reason in experience. I take Kant's thought as defining Enlightenment rational thought, at least for Hegel. And that as the basis for Liberal thought in that era. Universal values, intrinsic positive qualities of human nature, and founding knowledge and governance on rational materialist enquiry and principles. Submission of all to reason rather than power or religion, underpinning rule by consent, individualism, and liberalising trade and reducing government regulation of moral issues; individual's lives liberty and property secured by social contract. And perhaps most crucially, discourse about rights. Liberalism is a pretty vague term, and it's important to keep track of where and when a definition is being drawn.
Hegel called his approach Absolute Idealism: the progress of 'being' towards an ultimately comprehensible all-inclusive whole (the Absolute), which emerges from dynamic historical processes responding to necessity, developing through that into the increasingly complex forms of being we see. In this picture the state of comprehension of the Absolute is time-bound, there is a zeitgeist or time-spirit to it. His approach can be seen as basically apologism for the Prussian state of his era, and that interpretation has been popular for a long time, amplified by WW1 German imperialist militarism and WW2 fascism being linked to his thought, for instance by Popper in his book 'The Open Society And It's Enemies'. More modern scholarship has been reframing Hegel's thought as a communitarian critique of liberalism. Communitarianism is the idea that human identities are largely shaped by different kinds of constitutive communities or social relations, as opposed to a deeply individualist rational agents picture of human nature. Hegel arrived at picture recovering much of modern state enforced structure, but by very different rationale:
"If the state is confused with civil society, and if its specific end
is laid down as the security and protection of property and personal
freedom, then the interest of the individuals as such becomes the
ultimate end of their association, and it follows that membership of
the state is something optional. But the state’s relation to the
individual is quite different from this. Since the state is mind
objectified, it is only as one of its members that the individual
himself has objectivity, genuine individuality, and an ethical life.
Unification pure and simple is the true content and aim of the
individual, and the individual’s destiny is the living of a universal
life. His further particular satisfaction, activity and mode of
conduct have this substantive and universally valid life as their
starting point and their result.
"Rationality, taken generally and in the abstract, consists in the
thorough-going unity of the universal and the single. Rationality,
concrete in the state, consists (a) so far as its content is
concerned, in the unity of objective freedom (i.e. freedom of the
universal or substantial will) and subjective freedom (i.e. freedom of
everyone in his knowing and in his volition of particular ends); and
consequently, (b) so far as its form is concerned, in self-determining
action on laws and principles which are thoughts and so universal.
This Idea is the absolutely eternal and necessary being of mind."
-Hegel, Philosophy of Right
So, you could interpret that as challenging the idea of the state arising after the individual, by saying the individual can only be understood following participation in a particular state, and both developing progressively, into a whole which increasingly amplifies the capacities of both. His early essays 'The Positivity of the Christian Religion' and 'On The Prospects For A Folk Religion' are interesting clarification of how he aimed at a different balance of individual and community not only to the liberalism of his era, but also to Christianity of that time.
Communitarian critique of liberal individualism is something we can see as motivating Marx and Engel's Historical Materialsm. They saw this method as 'extracting the rational kernal from the mystic shell' of Hegel's picture of the dialectic, shedding the theism for a materialist picture. I am very sympathetic to Zizek's interpretation of Marx's thought as a kind of atheistic Christianity, seeking to develop from but hold on to Christian communitarianism through a rationalist critique.
"We again took a materialistic view of the thoughts in our heads,
regarding them as images of real things instead of regarding real
things as images of this or that stage of the absolute concept. Thus
dialectics reduced itself to the science of the general laws of
motion, both of the external world and of human thought — two sets of
laws which are identical in substance, but differ in their expression
in so far as the human mind can apply them consciously, while in
nature and also up to now for the most part in human history, these
laws assert themselves unconsciously, in the form of external
necessity, in the midst of an endless series of seeming accidents.
Thereby the dialectic of concepts itself became merely the conscious
reflex of the dialectical motion of the real world and thus the
dialectic of Hegel was turned over; or rather, turned off its head, on
which it was standing, and placed upon its feet." -Engels, in 'Ludwig
Feuerbach and the End of Classical German Philosophy'
That gives a good flavour I think of the impact of the shift from Hegel's picture of progress in understanding, in comprehension of the whole, to a picture of progress in material conditions and psychology as materially determined, making a kind of conspiracy-theory of history revealing to the initiated how society is inevitably set to become.
The big problems with these pictures of dialectic, both claiming to be refining rational and objective methods of interpretation if not enquiry, are underdetermination of causes, and subjectivity of conceptual units or supposed 'wholes'. Instead of being a way to understand the development of comprehension, or steps in the discourse of power, it becomes only a retroactive story-telling method taht narrows the picture of what could happen towards what did happen.
Popper actively sought to refine how we understand what science is, in order to demarcate Marx's 'materialist mysticism' out of it (along with Freudian Psychoanalysis) becuase it 'undermined and eventually lowered the traditional standards of intellectual responsibility and honesty'.
"The whole development of dialectic should be a warning against the
dangers inherent in philosophical system-building. It should remind us
that philosophy should not be made a basis for any sort of scientific
system and that philosophers should be much more modest in their
claims." -Popper in his essay 'What is Dialectic?'
A more humble view of a historical dialectic, can see it as a form of defeasible reasoning aiming to examine unconscious social motives or pressures, like understanding history in terms of a discourse of power derived from various factors (leverage, but that not always used by necessity in deterministic development), as say by Foucault.
I would argue we can in general look at progressive politics as hopeful about the future and the possibilities of social change, as a utopian tendency, and conservative politics as more fearful about social change and as motivated by fear of threats chaos and collapse. Jonathan Haidt relates the polarity to degrees of tolerance of ambiguity, and people's development or not of that to their sense of danger during adolescence, which could be impacted by things like participating in agrarian vs pastoral agriculture (herders can lose their whole wealth in one raid, & tend to have feuding & honour-cultures). This can recover a sense of dialectic development but an evidence based on, individualist vs communitarian, optimistic vs pessimistic, tolernace of ambiguity as grounded in a kind of secure-attachment and allowing scope for greater integration, and different locii of control.
The picture of liberal democracy as 'the end of history', eg by Fukiyama drawing directly on Hegel, is based in implicit historical materialism. This picture has grouped all the positives into that image of liberal democracy, and all potential for change in the negative image, into being causes of dystopia.
I would argue the relatively limited impact of leftwing politics in recent decades, has a lot to do with not having generated an image of a positive future, of a utopia to move towards, to challenge the status quo.
Christopher Hitchens, a Marxist, said:
“The problem of utopia is that it can only be approached across a sea
of blood, and you never arrive” in interviews. About which he elaborated "The search for
nirvana, like the search for utopia or the end of history or the
classless society, is ultimately a futile and dangerous one. It
involves, if it does not necessitate, the sleep of reason. There is no
escape from anxiety and struggle." – Love, Poverty, and War: Journeys
and Essays, 2004
I would say counterpoint to that is
"It is easier to imagine the end of the world than the end of
capitalism." -Mark Fisher in Capitalist Realism: Is There No
To see historical development as only incrementally away from what is undesirable, is I think to abandon the potential scope for human social development. While Marx's picture of history as necessitated developments towards communist utopia has almost discredited the idea of utopia, I would argue there is something we need to recover, about challenging the dichotomy between the current system, and change as inevitably into dystopia; that we need a positive image to move towards, to recover a faith that something more utopian is possible as a motivating image; a negation of negation, even in awareness of dystopian corruption of the drive towards such an image.
We need I think to restore faith in our capacity as societies for something in between incremental reactive change, and revolution. For me, an examination of history in terms of power dynamics and the impact of bargaining power of individuals and groups, can be a powerful source of insight to guide that; though it must be understood as defeasible reasoning, not the action of necessity. We need a challenge to 'There's no such thing as society', econimically, ethically, and psychologically. That means integrating individualism, with holistic and systemising reasoning, where the scope of individuals and of their societies are understood as deeply linked, and in a developmental discourse that can benefit both.