You are making a moral point I guess, which is: "why should we completely reject what is referred as prejudice, if it is a probabilistic guess based on statistical data".
I think I can see your point clearly.
Let's reformulate it for an event which does not require any morality, for example: the throw of a 6 faces unbalanced die. Let's assume the die is made in a way that you can not know the real probability of every number as an outcome, but you have the following statistics:
If you had to bet any money on the outcome, anyone that is able to see these statistics and know they are actually trustful, will point the money on number 4. No one will ever say: "Hey, what is this prejudice, we should also point money on other number because it is moral".
Let's translate the above back to the black/stealing example, and assume we have these statistics about thieves (i am giving random numbers):
With the same reasoning, for the next thief that will be brought to jail, one could make a bet on whether s/he is black or white, and based on the above statistics, I would bet on black.
So what about all these fighting prejudice?
That is probably another story.
There are many ways of interpreting the sentence "fighting prejudice", I will try and think about some of them:
First representative one is the one about misunderstandings and false assumptions. For example, let's say that you have the above statistics, and you randomly take a black person, citizen of the U.S.
Would you bet money on the fact that he is a thief?
If you would that would be a false assumption, because you assume that the probability of a black person to be thief is more than 50%, which can not be deduced by the above statistics. The percentage could be arbitrarily small (5% for example).
This is a common mathematical error that people easily do, and this kind of thing should be fought, because it is just mathematically wrong.
The second fight is against the mental process that changes a probability to 'certainty'.
This is more a psychological issue rather than a logical one, but it happens quite often that people, relying on the above statistics (let's even assume that the probability of a black being a thief is more than 50%), transform this statistical data into a full-generalization ("every single black person on earth is in reality a thief"). This should be fought as well because it is arbitrary.
Third fight to be done is a moral issue that might be claimed, which is:
Let's pretend you were born in a world where everyone who has the same skin color as you (let's say, green) is also a thief, except for a minority (say 3%) which includes you.
You, on the other hand, did never steal anything in your entire life and can not explain to yourself why other greens have to steal at all.
In this case:
- Is it morally correct to you that a person that meets you for the first time probably thinks of you as a thief?
- Shouldn't be it more correct that you were given the benefit of the doubt? If it were me, I would probably ask for a par-conditio (50%/50%), because it is not my fault that other greens steal, I am myself and I am extraneous to the process.
For this 3 types interpretation, I think "fighting the prejudice" is morally correct. I do not think though, that it can be achieved by completely throwing away useful statistical information, that can be always used to predict probabilities of similar events, and only for that.