We can definitely make a case for a materialistic explanation for what you are describing by thinking about entropy.
Classic philosphers held happiness to stem from the state of Ataraxia, a state of equilibrium and equanimity free from distress and worry. This state is difficult to attain and easy to fall from.
Simply put, there are much more ways to be unhealthy, hurt or unhappy than ways to be happy, fit and healthy. Many more configurations of the physical system that is you can be deemed undesirable or "unhappy" (injury, sickness, hunger, confusion, mental illness, social alienation, worry...) than there are who are desirable or "happy". Therefore statistically, if we don't put some work in the system from time to time it will most probably decay into one of the many undesirable states.
Just like houses fall in disrepair if left unkept, if we don't eat we get hungry, if we don't exercice we get unfit, if we don't take care about hygiene we get sick, if we don't contact our friends from time to time they drift appart, and if we contact them in one of the many innapropriate manners they run away, if we don't prepare carefuly for the future we are left vulnerable and open to worry. It's easy to cut one's limbs, it's practically impossible to put them back in place.
None of that requires some metaphysical moral drive to push us to work. Living, social beings are complex systems and complex systems require work to be kept together.