There are a few points to discuss here...
First, 'survival of the fittest' (in Darwin's model) is concerned with the fitness of a group or species, not so much the fitness of individuals. Any specific individual might meet its end by purely random events (e.g. volcanic eruptions). Darwin was concerned with how traits might spread through a population across generations, so that species adapt themselves to the environment.
Second, the concept of 'fitness' is intended to be ambiguous and value-free. Non-scientific people tend to impose human value judgements on the term, believing that those things we humans tend to value as individual strengths, skills, or competencies are necessarily more 'fit', when in reality — as nuclear weapons, anthropogenic climate change, sociopathic leaders, and the vast spread of global pollution show — our individual strengths, skills, and competencies might actually kill us off as a species (demonstrating a spectacular lack of fitness). Natural selection doesn't care whether the human species thrives, falters, survives, or dies; natural selection is at best an ex post facto 'I told you so'.
Third, the whole concept of fitness is far more subtle and complex than most people realize. People tend to think of fitness as a 'red in tooth and claw' matter: predators are more fit when they are stronger, smarter, and stealthier; prey are more fit when they are quicker, more agile, and more wary... But that's overly simplified. Many creatures — bees, ants, certain birds and fish, wolves, chimpanzees, etc. — have evolved social structures and group dynamics which increase the fitness of the group overall, even though each individual loses something to the group. There is a persistent thread in both philosophy and biology which tries to argue that human morality is itself a successful fitness strategy for our species: so successful, in fact, that it can accommodate certain amounts of 'unfit' selfish (immoral/amoral) behavior without threatening the survival of the species as a whole. Note that historically, large-scale efforts by the strong to squeeze out the weak — through expropriation, oppression, expulsion, or even genocide — are viewed with revulsion and disgust, and are often met with opposition, insurrection, revolution, or external attacks that decimate the putatively strong in defense of the putatively weak.
I'm not suggesting that we cannot or should not apply Darwin to human societies. But we cannot do so blindly or simplistically without ending up at some tragically poor theory (like Social Darwinism, biological eugenics, or 'clash of civilizations' type reasoning).