First, and I know this is somewhat subjective, I dislike the distinction between "soft" and "hard" science. When it boils down to it, all science is "soft" because at best we can falsify our theories. The issue with many social sciences is that it is hard to construct formal experiments or repeat an experiment, but we can still collect data to build more robust theories and to knock down old ones.
Now, as for application of hard sciences like physics and chemistry to "soft sciences" like psychology and sociology, the answer is fairly simple. It is because there is a sort of hierarchy from more foundational sciences to more derived sciences.
Physics drives chemical reactions. The reasons why certain molecules form, energy is released or absorbed during reactions, etc is all a matter of the physics of atoms. Biology is driven by chemical reactions. Thought is a manifestation, as best as we can tell, of the brain, a biological component of the body. With humans, behavior of individuals, couples together in social interactions in a meaningful way, and so our psychology and biology drive culture, group behavior, and so on.
Basically, while we might be able to understand the "what," such as "what do humans do when they interact" without these more foundational sciences, we cannot really understand the how or the why unless we look at the mechanisms which underlie them. So rather than being silly that "soft science" uses "hard science," it is necessary.
Example
This connection does not mean that one can easily throw any theory from physics at more derived fields of study. However, there are cases where there is a direct enough connection. For instance, evolution is essentially a stochastic process, modulated by environment and energy dynamics. There are fairly direct theories connecting thermodynamics to biology. Specifically, Jeremy England has proposed that looking at entropy within an open system (usually entropy is thought of in terms of closed systems) within an energy bath, it seems that "life" is really natural consequence of thermodynamics (Quantum Magazine).