It is not so much that the hypothesis is improbable if we see significant observations, but that the frequentist considers that the hypothesis merits rejection. If the distinction seems subtle, see if this explanation helps.
Bear in mind that significance testing is a frequentist method, not a Bayesian one. The difference is important.
The frequentist is not trying to attach a probability to a hypothesis. For the frequentist, a hypothesis is a statement about how the world is: it is either true or false and that's it. We don't know which, but that's our problem. Having chosen a given hypothesis, the frequentist designs an experiment and constructs a probability distribution for the data that one might expect to observe on the basis of that hypothesis. In the case of a null hypothesis, the hypothesis takes the form of a claim of independence. For example in a drug trial it might be something like: this drug has no effect, so we expect to see no significant difference between subjects who take it and those who don't.
The frequentist then gathers a data set. The data set is 'random' in the sense that it is unbiased, i.e. the method of gathering the data is independent of the hypothesis. The frequentist then chooses to accept or reject the hypothesis based on how significant the result is. Significance levels are usually expressed as a p-value. This represents the probability that you would get this data set, or a similar or more extreme one, on the assumption that the hypothesis is correct. If the result is highly significant, i.e. the p-value is very small, then the frequentist rejects the hypothesis. How small the p-value needs to be depends on the circumstances.
A common mistake is to suppose that the p-value is the probability that the hypothesis is correct. This is completely wrong. It is the probability of getting the observed data set, or a similar or more extreme one, on the assumption of the hypothesis.
The Bayesian looks at things entirely differently. The hypothesis is uncertain and therefore we assign it a probability, which represents our rational degree of credence in it. The data is then used to update this probability. To the frequentist, the hypothesis has no probability, but the data has a probability because it is drawn at random from a large set of possible data observations. To the Bayesian, the hypothesis has a probability because we are uncertain about whether it is true, but the data does not have a probability because we have the data, so it is certain. The frequentist and the Bayesian have opposite perspectives.
The two approaches may give similar results. If highly improbable observations are made, the frequentist rejects the hypothesis while the Bayesian conditionalises and has a posterior probability for the hypothesis much lower than the prior. But the two can also differ. If the observations are not significant, the frequentist just says: I have no reason to reject the hypothesis. This does not mean there is a reason to accept it, just no good reason to reject it. The Bayesian happily updates on the data either way and comes up with a new posterior probability. There are other differences too: e.g. Bayesian methods treat nuisance parameters differently.
Bayesians usually prefer to use Bayes rule in its ratio form to compare rival hypotheses. Often we cannot say categorically that this data makes this hypothesis probable or improbable, only that it favours this hypothesis relative to some alternative. This seems to be what Sober is referring to with his example of the two urns.
To your final question, I would say that under some narrow conditions it may be OK to reject a null hypothesis based on significant data, but only when no alternative to rejecting it is available. If the drug trial shows that all 1000 of the patients who took the drug got better, while all 1000 who didn't died, that's pretty good evidence that the null hypothesis can be rejected. We might still want to check carefully that there are no confounding variables and the data set was completely unbiased and nobody was fiddling the results. But a highly significant result like that is pretty good. What alternative explanation for the result is plausible?
However, it is true to say that null hypothesis testing is overrated and overused. You might care to consult the book: The Cult of Statistical Significance by Ziliak and McCloskey. There is also a short paper by Amrhein, Greenland and McShane, published in Nature 567, 305-307 (2019), in which the authors, together with more than 800 other signatories, call for a restriction on the use of significance testing and confidence intervals. Speaking of confidence intervals, they are also frequently misunderstood and misused.