Since Buddhism denies the existence of the soul, in the theory of no-self, what is liberation or nirvana in Buddhism? Who attains nirvana? Who experiences cessation of suffering?
There is no "who", that is the whole point. Nirvana is a state in which the self ceases to intrude and is revealed as an illusion.
One might rather ask, "what attains Nirvana? What experiences cessation of suffering?"
When asked this, the current Dalai Lama of Tibetan Buddhism replied cheerfully, "I don't know!"
I have seen it described as your "Buddha nature" or even your "dharma", but these are not ideal terms and some people do not think it appropriate to use them in this way.
Allow me a problematic analogy for a moment. If we are using a computer, and we switch from a text processing app to an image processing app, has the essential nature of the computer changed? What if we switch from MacOS to Windows? In each case the computer appears different, performs different behaviors, asks for different user actions, but...
What is the 'is-ness' of a computer?
I dislike making analogies between computers and the mind — people tend to let that analogy carry them away in odd directions — but the point is that much of what we typically consider to be the human 'self' is surficial and retrospective. We all have behavioral patterns, skill sets, beliefs, etc., and when we think about each other (or about ourselves) what we think about are those patterns, skills, beliefs, and whatnot. Joe is a doctor; Amy is temperamental; Bill can't walk and chew gum at the same time; Mary is extremely articulate: these kinds of things are elements of our identified selves. But what if these things change? What if Joe can no longer do medicine, or Amy becomes peaceful, or Bill trains as a world-class gymnast, or Mary develops aphasia? What happens to the 'self' then?
What we conceptualize as the 'self' is largely (as in the computer analogy) a set of interfaces with other beings in the world. We choose to be this kind of a spouse, or that kind of a parent; to work in one career or another or none; to throw ourselves into the world with abandon or walk carefully around the edges. The way we interact with others gives us identity and a sense of self. These identities and senses-of-self help us coordinate, so that (for instance) I can say "Please take this thumb drive to Wilma; she'll know what to do", and that set of words will accomplish something, because you know who I mean by 'Wilma' and Wilma has developed the identity (for herself and us) of someone who 'knows what to do'.
When we stop using the interfaces on a computer — and this is where the analogy goes to pot — it stops being anything. It never had a sense of self-identity (it doesn't 'know' whether it's acting as a text or image processor), and if we're not interacting with it we're not giving it an identity, so in a sense it ceases to be. But it's not the same for people. We have strong self-identities that drive us to interact with others in our established ways. Even if we are absolutely alone, our minds will turn over looking for ways to meaningfully express our identities: our 'selfness'. We don't just 'turn off' the way a computer does.
Now, to answer your question directly, part of the Buddhist programme aims at discovering what lies underneath all the surficial, retrospective interfaces that constitute the identified self. That is the spirit of meditation. During meditation one doesn't interact with others, one doesn't accomplish any ends, one doesn't enjoy the peace and quiet, one doesn't let the mind wander from this to that interesting thought or fantasy. Meditation means sitting and experiencing the inner world not as an interface to the outer world, but as a connected whole with the outer world. The identified self (eventually) melts away for a bit, and what's left is what we want to understand: the self-without-identity (Buddha nature). Think of Sartre's being-for-others, being-for-itself, and being-in-itself. In Buddhism, the identified self is (more or less) a composite of the first two, and the goal is to reach an understanding of the last.
The self is not a thing independent from the rest of the world. It might be helpful to think in terms of 'You' and 'you' (or 'I' and 'i'). 'You' is male/female/other. 'You' is your job, your politics, your likes and dislikes. 'You' is the grudge you can't get over and all the experiences (big and small) that shaped your perception of the world and your place in it (and apart from it). On the other hand, 'you' (small y) is what's left when you peel away all the accumulated layers of 'You'. As you learn to let go of big Y 'You', as you catch glimpses of the real 'you' that is connected collectively with everyone and everything else, 'you' begins to attain enlightenment. Similarly, as my attachment to 'I' becomes less, 'i' begin to attain enlightenment.
Buddhism does not deny the existence of a soul, an Atman. This is a common misconception. The practice of Anata (not self) is to search all of what can be perceived or conceived, conditioned reality, labeling everything as “not self, not self”. But there is also the Unconditioned, the unmanifest, the unborn, which cannot be spoken of. Outside of the five skandhas and all of this reality, but still there. Anata does not mean, “There is no soul; realize this fact.” It means, “Not self” or “Not soul” as in, “This is not self.” Here is one scholar clarifying this: https://youtu.be/FEnb2cFWKBs
There is also a story of the buddha being asked if God exists. He said that claiming, “God exists.” is untrue, and saying, “God does not exist.” is untrue. Yes that’s illogical. The Buddha has been painted as a monist, he was not a monist. He believed in a transcendent reality that is achieved by utterly letting go of this one (by seeing every phenomena in this one is impermanent, impersonal, and as causing suffering when clung to).
I'd like to offer an alternative perspective. Consider the evolution of the soul: that which supposedly eventually attains nirvana.
Consider in practice one Charles Derringer Smith, a successful retired lawyer and life-long agnostic. He's well past retirement age but in good health, living comfortably in an elegant house in an upper-class suburb, supported by his well-structured pension as well as several children and grand-children whom he seldom sees, and has little real knowledge of. His loving wife is still by his side, takes care of his few daily needs, but spends most of her time at the bridge club, as does he at various upper-class social venues with his remaining long-time friends.
Then, TRAGEDY strikes. Following the advice of his long-trusted personal medical practitioner, he accepts two vaccinations for the currently popular seasonal pandemic, contracts severe blood clots, is soon laid out on a bier, and so kicks his personal bucket far down the road. More practically, he cashs in his chips.
He then finds himself "on the other side". MOST amazingly, NOT DEAD!. Well, sort of. In this new state of existence he can drift about wherever he pleases, and soon discovers that he can "visit" his loved ones, even not-so-loved ones; but none can see him, or are they the least aware of his presence and continued existence. And it doesn't take long to realize, by listening in on conversations, that their opinions of him are at best indifferent, and often contemptuous.
Which prompts deep and painful reflections, long overdue, on just WHO Charles Derringer Smith really is. Or was. And, in any case, does it matter? And WHY?
A few months later, the late Charles Derringer Smith is painfully and most certainly aware that whomever he WAS is not something to be proud of, has largely been forgotten by his benefactors, once-friends, and not-so-dearly-remembered relatives, whose remembrances of him, as expressed in conversations of chance overhearing, are directly proportional to their personal portion of his will and legacy; financially speaking. Other than that, he has all but been forgotten, even by those who openly despised him and were most grateful for his demise.
After having reflected on these sorry revelations for a number of increasingly shameful years, the former Charles Derringer Smith is visited by an "angelic helper" who advises him, kindly but firmly, that he will soon need to reincarnate, lest his astral vehicle go into permanent decline. He then risks losing his essential nature as a potential human being, becoming a fading shade, a demented ghost, and at the last a mere dying shriek on the wind. And that would be it. Not only Charles Derringer Smith, but that which gave birth to him, will then be no more; nor ever was. Nor ever will be.
So what is it, then, that eventually DOES reincarnate?