This is not a simple affirmative.
First, we have to understand exactly what you mean by "set". There are many Set Theories, with different objectives.
Second, categories are not necessarily sets. This is a very complicated topic; someone could argue that it makes no sense to talk about categories that are not sets, but the language of categories is in principle independent of a set theory (but uses some intuitive notion of "collection").
Here I can make a point. There is a difference between the language that we use to talk about math and math. For a realist, math has some reality that does not depend on the language we use to express that reality. So, according to this view set theory and category could be seen as just "ways" to express that reality. When you say a "number" is a "category" then we have to be clear if we are talking about numbers as "real things" in some sense that coincide with categories as "real things" in that same sense, or if you are saying just that a number is a linguistic entity that is referred to by the language that we use for categories.
For a Platonist, there is a whole world where math exists and from that world we just experience its shadow. For an Aristotelian, math exists in the things, in some sense, there is no other world necessarily. So, if the ontological status of math is some kind of realism, then you could have in the own reality of math a problem to say that "a number is a category", you could linguistically say something like that, but not in reality. If the approach to the question pressuposes some kind of nominalism, and you think that math doesn't really exist, is just a linguistic entity, then you could say something like that.