The word "math" is here ambiguous. As a matter of fact (pun intended) Hume distinguished between (1) arithmetic and algebra, which are, according to him, based on relations of ideas, (2) geometry, which is based on matters of fact, but is relatively certain and reliable, and (3) other matters of fact.
Concerning arithmetic and algebra:
It appears, therefore, that of these seven philosophical relations, there remain only four, which depending solely upon ideas, can be the objects of knowledge and certainty. These four are RESEMBLANCE, CONTRARIETY, DEGREES IN QUALITY, and PROPORTIONS IN QUANTITY OR NUMBER...
There remain, therefore, algebra and arithmetic as the only sciences, in which we can carry on a chain of reasoning to any degree of intricacy, and yet preserve a perfect exactness and certainty.
Concerning geometry, Hume reiterates the old doubts about the axiom of parallels, doubts that were to be fully justified about a hundred years after Hume, with the development of non-Euclidean geometries.
I have already observed, that geometry, or the art, by which we fix the proportions of figures; though it much excels both in universality and exactness, the loose judgments of the senses and imagination; yet never attains a perfect precision and exactness. It's first principles are still drawn from the general appearance of the objects; and that appearance can never afford us any security, when we examine, the prodigious minuteness of which nature is susceptible. Our ideas seem to give a perfect assurance, that no two right lines can have a common segment; but if we consider these ideas, we shall find, that they always suppose a sensible inclination of the two lines, and that where the angle they form is extremely small, we have no standard of a right line so precise as to assure us of the truth of this proposition.
But geometry is still, according to Hume, accurate and reliable relative to knowledge of other matters of fact, because it depends "on the easiest and least deceitful appearances".
Though geometry falls short of that perfect precision and certainty, which are peculiar to arithmetic and algebra, yet it excels the imperfect judgments of our senses and imagination ... since these fundamental principles [of geometry] depend on the easiest and least deceitful appearances, they bestow on their consequences a degree of exactness, of which these consequences are singly incapable.
(The quotes are from A Treatise of Human Nature "Of Knowledge")