I would speculate that it has to do with the (apparent) indispensability of numbers.
To unpack that a bit, I want to begin by talking about what it would mean for the word "dog" not to exist. Taking the word "dog" as a sign, let's divide it into signifier and signified, taking care to identify both as essentially abstract. What I mean by that is that the signifier "dog" is not the sound my lips make when I say "dog," nor the images of the letters 'd,' 'o,' 'g' on the screen before you, but the sequence of phonemes /d/ /o/ /g/ within the abstract structure of the English language. And likewise, the signified of "dog" is not any physical dog in the world, but the abstract concept of dog. So if we say that the word "dog" doesn't exist, then we're either saying the signifier or the signified of "dog" doesn't exist.
The existence of signifiers; the existence of concepts
Now, I think that to say that the signifier "dog" doesn't exist is tantamount to saying that some number doesn't exist! That's because if numbers exist, then sequences of discrete symbols exist; and so if sequences of discrete symbols don't exist, then numbers don't exist. So in that case, by worrying about whether the signifier "dog" really exists, then we're already worrying about whether numbers really exist. I think these two worries are close enough that they come and go together.1
That means that this question is really about the signified -- the concept of "dog." (This justifies Mauro's move towards concepts in his answer.) Now, obviously it would be wrong to say that the concept "dog" itself doesn't exist. It plainly does! We use the word dog, we think about dogs, we think of things as dogs, and so on. So it wouldn't make sense to worry about that. If we worry that the concept "dog" doesn't exist, then our worry must be that the concept "dog" is in some sense just made up -- that it's just in our heads, and it doesn't correspond to anything real. In that case, we could just skip past the concept, which is just made up, and go straight to the set of actual, physical dogs-in-the-world. That would make the concept "dog" a mere shortcut for referring to those dogs. This is the nominalist position.2 So the fundamental question you're asking is why it's harder to accept nominalism about numbers than it is to accept nominalism about dogs.
The problem with nominalism about numbers
A tempting answer is to say that it's obvious that we have dogs here before us, and so we can easily throw out the concept and just focus on the physical dogs that confront us. But we can't do that so easily with numbers; it's not obvious that any physical thing in the world confronts us with fiveness. It's true that we can find objects that come in fives, but have we really found fiveness in those objects, or have we structured our understanding of those objects using the concept five? As soon as we conceive them as discrete objects, we've given them a number. Couldn't we just as easily see five pebbles as ten half-pebbles? So it seems that the fiveness isn't in those objects after all.
Now, we could say the same thing about dogness -- dogness isn't really in dogs. After all, we don't have to divide the animal world up into species. There are many other coherent ways to construct groupings of animals that would help us to refer to the particular animals in the world. And we don't even have to use groupings of animals; we could think about groupings of ecosystems, or we could be reductionists and talk only about subatomic particles.
But note the difference between these two moves. When we throw out the concept "dog," we can use any number of other kinds of concepts to replace it with. But when we throw out the concept five... we can only replace it with the concept of another number! In our description of pebbles, we could do away with the concept of five only by introducing the concept of ten. There seems to be no way to do away with numbers entirely.
I should mention, finally, that this line of reasoning is (I think) closely related to the Quine-Putnam indispensability thesis, which I will playfully summarize as the argument that because we worry about the existence of numbers, we ought to go ahead and believe that they exist! However, there's a very interesting book called Science Without Numbers that attempts to show that numbers might not be indispensable after all -- hence the "(apparent)" in the first sentence.
1. Note that by worrying about whether numbers really exist, we're not necessarily worrying about whether the signifier "dog" really exists, because numbers have additional structure; they can be added, multiplied, and so on, while signifiers can't be. But if we say the signifier "dog" really exists, then we might as well go ahead and add the properties to "dog" necessary to treat it as a number, because we've already committed ourselves to the existence of an abstract object. So when we wonder whether numbers exist, we might as well be wondering whether the signifier "dog" exists; I think there's no good reason to reject the existence of numbers while accepting the existence of signifiers.
2. This line of reasoning applies to signifiers as well; when we worry about the existence of signifiers, we aren't worrying that they don't exist in our heads -- they obviously do! We're worrying about whether we've just made them up or whether they have some real correspondence to something outside our heads other than all of the physical "dog" sounds we make with our mouths and physical "dog" shapes that we draw and type. But again, in this case, it's clear that this worry is almost the same worry as that about the existence of numbers.
