Nietzsche believes that modern human beings were alienated from themselves. what was the nature of this alienation? how did we come to be alienated? what exactly are we alienated from?
(I believe this is in the Genealogy of Morals)
I wouldn't say that Nietzsche's critique of modern man and his alienation is exclusive to On the Genealogy of Morality. In fact, it would be hard to systematically delineate exactly where Nietzsche discusses and delimits the nature of alienation.
One thing I think I can say with certainty is that Nietzsche does not associate alienation solely with modern man, but rather, as Marx did similarly, alienation is a fundamental fact of human existence that can be beautifully captured in Nietzsche's quasi-thought experiment of the eternal return (an idea that had been transposed more or less accurately to the literary work of "The Unbearable Lightness of Being").
Essentially what the eternal-return thought experiment evokes is that sense of lightness inherent in a life that is transitory and passing, a life that is merely a prelude to an ethereal afterlife; this lightness is not particularly anchoring; conversely, those that accept the eternal-return doctrine should feel more inclined to pay attention to this life as they will inevitably live every single act and event over again and so on for eternity.
Alienation thus follows for those, like many religious people, who are estranged from this life because it is only a "veil of illusions" or "superficial and fleeting". Alienation is then a characterization of a specific psychological type, not something that "becomes" but perhaps something that "is" an inherent property of a psychic predisposition.
While Nietzsche does not employ the word alienation as a standard term in his discussions and critiques, he does discuss it rather in depth, albeit obliquely and sometimes indiscernibly, and it is inextricably linked with many of his key concepts (such as "amor fati," "the eternal-return," "ressentiment," etc.) Daybreak may be a rather illuminating text to look at to view Nietzsche's delineation of psychological types and morality and alienation.