What does Nietzsche mean, in the Genealogy of Morals, when he refers to European Buddhism? Did he think Europeans misunderstood Buddhism?
I disagree with Jo Wehler's answer... I think.
At least, the quotation Jo uses from the preface to the Genealogy of Morals doesn't seem to support his conclusion for a few important reasons.
That quotation contains a few references to bigger ideas Nietzsche developed. Nietzsche loathed nihilism and mediocrity in equal measure. He characterised Christianity as a "slave religion" because it glorified submission, weakness, and mediocrity, he thought it thus inhibited human potential. He didn't like Buddhism either, because he reckoned it fundamentally nihilistic.
It is possible to read this, especially given "Will to Power", to mean Nietzsche hated weakness or the weak generally. But a more nuanced interpretation finds it more a question of what values individuals should aspire to. In his view aspiring to weakness was a social problem, as was glorifying the humble and last-will-be-first ethos of Christianity and Buddhism.
These two themes must be understood within the context of the "God is dead" problem. That is, what values does society adopt when they've "killed" God? Or when society is post-religious and morals are not driven by old religious certainties.
Nietzsche was worried that a post-religious society would risk falling into bad habits, where people were driven purely by herd instincts which again, would promote little more than nihilistic mediocrity. This seems to be the crux of the "European Buddhism" term. More specifically, the influence Buddhism was having on European philosophers of his time, which empowered nihilistic thinking. While initially influenced by Arthur Schopenhauer (thanks CriglCragl), he later grew to reject Schopenhauer's rather bleak philosophy.
Schopenhauer's work was an attempt to blend western and eastern thought, which produced philosophical pessimism. He took a lot from Buddhism, and it seems he was also very fond of the [Hindu] Upanishads. Schopenhauer concluded upon the idea of a universal 'will' which bends life toward suffering. So Nietzsche's notions have to be understood in that context.
The answer seems that Nietzsche thought Buddhism influenced contemporary European philosophy, and in doing so empowered nihilism and made European thought more Buddhist as he understood it. "European Buddhism".
Nietzsche in the preface to the Genealogy of Morals, section 5:
The most specific issue was the worth of the “unegoistic,” of the instincts for pity, for self-denial, and for self-sacrifice, of things which Schopenhauer himself had painted with gold, deified, and projected into another world [verjenseitigt] for so long that they finally remained for him “value as such” and the reason why he said No to life and even to himself, as well. But a constantly more fundamental suspicion of these very instincts voiced itself in me, a scepticism which always dug deeper! It was precisely here that I saw the great danger to humanity, its most sublime temptation and seduction.—But in what direction? To nothingness?—It was precisely here I saw the beginning of the end, the standing still, the backward-glancing exhaustion, the will turning itself against life, the final illness tenderly and sadly announcing itself. I understood the morality of pity, which was always seizing more and more around it and which gripped even the philosophers and made them sick, as the most sinister symptom of our European culture, which itself had become sinister, as its detour to a new Buddhism? to a European Buddhism? to—nihilism?
Nietzsche hated any form of compassion, his ideal was the strong man. Buddhism became known in Europe during Nietzsche's time. Buddhist ethics emphasizes compassion. Therefore Nietzsche rejected Buddhist ethics alike rejecting Christian ethics.
Nietzsche does not think that Europe misunderstands Buddhism.
It is very likely both Nietzsche and other Europeans all misunderstood Buddhism at the time. The Palli cannon has be translated but a majority of Buddhist Mayhana texts have yet to be translated still into European languages. Much of the error in Nietzsche idea of Buddhism was the mistranslation of the concept of Sunyata into the word "nothingness". This more commonly gets translated now as emptiness, because the meaning is closer and does not have the false connotations of the word "nothingness". Sunyata means something more akin to "without permanent aspect".
The translations and actual literature about Buddhism available at the time were sparse and poorly translated. This colors all of Neitzsche's conceptions about Buddhism with a false understanding. This is not really a fault of Nietzsche. He just didn't have any good source material.
I have practiced Nichiren Shoshu True Buddhism since 1974. Most Philosphers draw from history. Nietzsche was no exception. Unfortunately for many their knowledge of Buddhism was limited. The concept of Nihlism is attributed to Zen, which most Buddhist scholars do not even consider Buddhism because Zen follows NO DOCTRINES. Nihilism is NOT BUDDHISM.