I heard that there are philosophy classes that teach philosophy in a very rigorous and formal and mathematical manner. Are there books that do this? I would like some recommendations.
Check out this course on Coursera:
There is a particular school of philosophy, called "analytic philosophy" which aspires to formal rigor, and there is a subject area within philosophy, called "formal logic" which is very closely related to mathematics and is often taught in a mathematical manner. Researching either of those topics should lead you to a wealth of good sources for what you are looking for.