I would not myself use the term 'racism' here if the real grounds of complaint and aversion are as you say aesthetic : A-citizens object to the presence of B-citizens on aesthetic grounds of appearance. To my mind, this doesn't entail 'racism'. 'Racism' involves a spectrum of attitudes that are missing from your question : if As believed Bs to be culturally inferior, or intellectually inferior, or morally inferior, or if Bs were objected to purely as ethnic immigrants intent on exploiting country A's wealth, then talk of racism would be appropriate. But none of this comes up in your example. I can, however, see what you had in mind, or think I can : certain ethnicities are rejected on grounds of their appearance, particularly colour, or on points of physiognomy. This is 'racism', or an example of racism, as the term is widely used. On this basis there is no disagreement between us.
Act-utilitarians contend that the rightness or wrongness of an
action is determined by the consequences of that action alone.
The right action for any agent is allegedly that option open to
him that will produce the best overall result' [in a particular situation : GT]. (Donald C. Emmons, 'Act vs. Rule-Utilitarianism', Mind, New Series, Vol. 82, No. 326 (Apr., 1973), pp. 226-233 : 226.)
If we consider the situation in act-utilitarian terms, I don't see how a utilitarian calculation can be made. How can you calculate the total disutility that accrues if A's are compelled to accept the aesthetically objectionable Bs as against the utility of better-living conditions that would accrue if Bs were let in - also including in the calculation the disutility to Bs of knowing that their presence is resented ? Aesthetic objectionableness, better-living conditions, awareness that one's presence is resented - these are incommensurables. By this I mean that there is no general lexical ordering between them. One has to decide how much importance to give them, individually and in relation to one another, from one occasion to another; a decision is not a calculation. Use of single-metric terms such as 'utility' or 'happiness' creates the appearance of a common metric when often there is none, as I suggest in the case you describe. See Joseph Raz, 'Value Incommensurability: Some Preliminaries', Proceedings of the Aristotelian Society, New Series, Vol. 86 (1985 - 1986), pp. 117-134. Or, longer but easier to read and just as good, John Gray, Isaiah Berlin: An Interpretation of His Thought, ISBN 10: 0691157421 / ISBN 13: 9780691157429. Published by Princeton University Press, United States, 2013.)
I don't see, by the way, that if we are going to use the notion of utility the strict logic of utilitarianism allows us to exclude 'nasty' utility. Whatever can yield utility - say, as many have done, whatever satisfies a preference - has a proper a place in an ethical theory that derives its entire rationale from utility. Not nice utility, just utility. This may, indeed, open the way to a serious critique of utilitarianism. But that is not my present concern. Add it to the problem of incommensurability, however, and the critical questions pile up.
An act is right if and only if the consequences of everyone's doing that type of action in similar circumstances would produce the best overall result.
Turning to rule-utilitarianism, we should have to consider whether on balance, taking one occasion with another, having a rule that rejects immigration on grounds of an ethnicity's appearance, particularly colour, or on points of physiognomy, will produce a better overall result than having a rule that does not reject immigration on these grounds or having no rule at all. Here we can only guess, not calculate : my guess is that the overall result would be better if there were not a rule that rejects immigration on grounds of an ethnicity's appearance, &c. This is based on general considerations about the mutual benefit of immigration but this takes us into economics and sociology, and beyond philosophy.