The enterprise of reducing mathematics to logic is called Logicism. There have been two different goals in this enterprise, the first is to reduce just arithmetic of natural numbers, which in many ways is the easiest and most basic part of mathematics. The other is to reduce all of mathematics, or at least as much as we can, to a set of axioms (consider something like ZFC set theory).
Logicism, historically, was lead by Frege, Dedekind, and Peano among others. Peano is most famous for giving a set of axioms that allow for the construction of the natural numbers and arithmetic, while Dedekind is arguably most famous (at least in terms of this conversation) for discovering what are called Dedekind cuts, partitions of the rational numbers which allow for the construction of real numbers. Frege set out to just form a logical foundation for arithmetic, while also developing the tools and languages of modern logic in the process:
In his book of 1879, Begriffsschrift: eine der arithmetischen nachgebildete Formelsprache des reinen Denkens, he developed a second-order predicate calculus and used it both to define interesting mathematical concepts and to state and prove mathematically interesting propositions. However, in his two-volume work of 1893/1903, Grundgesetze der Arithmetik, Frege added (as an axiom) what he thought was a logical proposition (Basic Law V) and tried to derive the fundamental axioms and theorems of number theory from the resulting system. Unfortunately, not only did Basic Law V fail to be a logical proposition, but the resulting system proved to be inconsistent, for it was subject to Russell’s Paradox.
Logicism has been subjected to a number of set backs, some so devastating that many philosophers believe that the program cannot ever succeed. One of them is the weakness of second order logic and how Frege's Basic Law V failed to do what he had hoped it would, derive all of arithmetic. Even if the law were not subject to Russell's paradox, it still has a glaring ontological issue: the Julius Cesar problem. In essence, the problem says that something like Hume's principle (which is very similar to Basic Law V) cannot provide enough of an epistemic reason as to why we should pull numbers out of our arbitrary definitions:
[Gl, §55:]
… but we can never – to take a crude example — decide by means of our definitions whether any concept has the number Julius Caesar belonging to it, or whether that conqueror of Gaul is a number or is not. [from the Austin translation in Frege 1974]
Ultimately, many believe Frege's original conception of logicism is not a paradigm that can work (although there is renewed interest, see neo-logocism).
There are two, maybe three, other glaring problems for reducing mathematics to logic. They come in the form of Gödel's two incompleteness theorems and Tarski's undefinability of truth.
Peano's axioms turn out to be subjected to Gödel's two incompleteness theorems. The first theorem (to be brief) states that any formal system, meaning a set of axioms within a deductive logic, is subject to what are called Gödel sentences. These sentences that are true (semantically) but unprovable (syntactically) within the system itself. Gödel accomplished this theorem by giving a generalized way to construct these sentences in any formal system that is strong enough to formalize Robinson arithmetic (a weaker form of arithmetic than Peano's axioms). Presburger arithmetic is a formal system of arithmetic that does not contain multiplication and is not subjected to Gödel's first incompleteness theorem.
This brings up something important. What is really of note here is that, Gödel's result is not necessarily about mathematics itself (unless you believe that mathematics is strictly logic in disguise); in essence, his results are about the formal systems themselves, or the logic itself. The results are syntactical, meaning that they are about the syntax, the rules of the system, and not about the semantics, the meaning we ascribe to the system. Formal systems are just a bunch of arbitrary rules and symbols until we give them meaning, and Gödel's results basically say "If you have this set of rules or anything stronger than it, you can derive this set of sentences." It doesn't matter what meaning we give to them, that the system is talking about numbers and arithmetic. Before we introduce any sort of semantic content to the theory, the Gödel sentences will still be there because they are constructed in a purely syntactical way. This is a major issue for logicism, because it shows that any sort of logical system will have true statements about the natural numbers, statements we know are true, but are unprovable in our theory. Logicism does not abide that.
Gödel's second incompleteness theorem says that "no consistent set of axioms (with enough strength) can prove it's own consistency." Another way to read it is that "any, sufficiently strong, axiom system that can prove its own consistency is inconsistent." The two results combined say that "Any consistent system with at least enough strength to define Robinson arithmetic cannot be complete due to its Gödel sentences and it cannot prove its own consistency." The system might be consistent, but we cannot formulate a proof of that within the system itself.
As a result of these two theorems, we, as mathematicians and philosophers, have to treat our axiomatic systems with caution. Many believe that Peano arithmetic and ZFC set theory are consistent, however we only believe that because we have been spending so many years studying them and have yet to find an inconsistent sentence. We make that judgement purely due to induction (and a little bit of meta reasoning, it seems very unlikely that ZFC is inconsistent) but we have no formal proof within the systems themselves that they are consistent. You can, however, chain consistency proofs of smaller, weaker systems but this only pushes the epistemic problem of consistency higher up the chain. ZFC can prove the consistency of PA; a different set theory called NBG is consistent if and only if ZFC is consistent; an even stronger set theory called MK can prove the consistency of ZFC; and so on. However, at each step we have to assume that the higher theory is consistent in order to believe the consistency proof of the lower theory. Eventually, the systems we get to are so big that we cannot have reasonable assumptions about their consistency. Due to Gödel's second incompleteness theorem, there will never be a ceiling theory that can prove its own consistency and can be used to prove all of the weaker theories.
Tarski's undefinability theorem is very similar to Gödel's theorems, except that it focuses on the definability of truth. Just like how Gödel shows that a formal system cannot show its own consistency, Tarski shows that a predicate that defines when a sentence is true cannot be formulated within a system itself and must come from some other, "meta" system. The study of metasystems of this kind, the kind that allow for a definition of truth, is called model theory, which Tarski spearheaded. His results, though, ultimately show that there can never be one theory that contains the means to show that its sentences are true because the "truth" of its own sentences can never be defined within itself.
So, the enterprise of Logicism has run into a few snags. Some people, namely neologicists (myself included), believe that the results of Gödel and Tarski don't necessarily preclude an ultimate reduction of mathematics to logic; more so it is believed that there will always be a few things left out of the theory and some assumptions we have to make but cannot prove. The Stanford Encyclopedia of Philosophy article linked in the beginning of this answer adds a lot more detail to the philosophical objections to logicism theories and is worth reading in order to obtain a better grasp of this subject. All this being said, the consistency of ZFC is widely believed to be true and mathematicians have no problem, in practice, of trusting this assumption. It is almost universally considered a true mathematical proof if you can formulate your theorem as a derived sentence of ZFC. There are some logicians who don't believe ZFC is consistent, however. The recently passed logician and mathematician Jack Silver was very vocal about his opinion that ZFC is inconsistent and he worked vigorously to construct a proof; however, he was unsuccessful.
Ultimately, many believe that it is not possible to entirely reduce all of mathematics to logic, given Gödel and Tarski's results. Some still do; however, as of this moment, no logicist program has done exactly what those like Frege and Peano wished it would do.
