Quine's short 1954 paper, Quantification and the Empty Domain (Journal of Symbolic Logic, Vol 19 Number 3 September 1954), addresses this.
Quines two reasons are:
If some first-order statement is true in every domain larger than a domain D, then it is true in D. But this is only true if D is not the empty set. Philosophically, we want our statements to be true of all sufficiently large domains of discourse, but this theorem says that including finite ("small") domains of discourse does not change things. However, including the empty domain of discourse would change things.
It's easy to tell anyway whether a first-order statement is true for the empty domain. So it seems unnecessary to include this in our theory when we can just handle it as a special case.
I would summarize this to say that there are clearly practical reasons to exclude the empty domain; this isn't to say that these reasons are natural, but I would guess Quine is more concerned about what is practical than about what is natural.
In R. M. Martin's 1965 Of Time and the Null Individual, it is actually proposed that we introduce a "null individual" to our logic. As best I can tell, the effect of this is that the empty domain can be discussed, and that essentially the same logical theorems hold true for the empty domain as held in classical first-order logic. Whether you consider this to be justification for leaving the empty domain out in the first place is another matter. Perhaps the "null individual" is little more than an artificial way of making the empty set behave as if it were nonempty. But perhaps it makes sense philosophically to say there must always be a null individual x for which statements such as Px and Qxx can be written, even if we do not assume that this null individual actually exists; in this way we avoid the problems the empty domain could pose for logic, but at the same time we avoid assuming a priori that at least one thing exists (for the whole point of the null individual is that it is not considered to exist).
There is some other discussion in A Note on Truth, Satisfaction and the Empty Domain, Timothy Williamson, 1999.