What is a mathematical or logical name for the process of proving a statement by exhausting the domain?
The general term is unsurprisingly proof by exhaustion. From WP:
Proof by exhaustion, also known as proof by cases, proof by case analysis, complete induction or the brute force method, is a method of mathematical proof in which the statement to be proved is split into a finite number of cases or sets of equivalent cases, and where each type of case is checked to see if the proposition in question holds.1 This is a method of direct proof. A proof by exhaustion typically contains two stages:
- A proof that the set of cases is exhaustive; i.e., that each instance of the statement to be proved matches the conditions of (at least) one of the cases.
- A proof of each of the cases.
As for your scenarios and additional questions, it's best to see the two scenarios as using the same form of inductive logic to draw relatively certain conclusions about the contents of the bag by empirical means. In one case, you are trying to show no balls are a certain color, and in the second, all balls are a certain color; neither can be done with deductive, mathematical rigor. However, by conducting millions or billions of trials, one can gain an epistemologically sound confidence in an admittedly fallibilistic conclusion. One often see such confidence in mathematical conjectures, such as in Goldbach conjectures that are shy of proof, but may yet be shown to be false which was the fate of Fermat's primality test.
Argument: There are no red balls in the bag.
Action: Pick up balls at random until you find a red one.
Action: Pickup a random ball from the bag.
Argument: All the balls in the bag are of same color as the ball you picked up.
If one is trying to prove there are all or no red balls, proof by exhaustion will provide, as noted in the other answer, an inductive proof, but not a deductive one in the scenarios above. This is because in a sense, one cannot truly exhaust the problem space. However, after drawing a billion times from the bag, an inductively cogent (not a deductively sound) proof, can be said to exist. Of course, scientists accept inductive proofs, whereas mathematicians tend to reject them. There are exceptions, of course. Computer scientists are quite comfortable with probabilistic primality tests such as the Miller-Rabin test.