I'm not aware of anything that's accepted that is exactly what you want, but I will note that Bayesian reasoning is a framework that is great at modeling exactly what you want.
See the Wikipedia entry, for example.
In particular, you can include your inference steps as part of the model. So a valid proof of A
assuming B
would have p(A|B) = 1
; if you have any invalid step you can swap in the probability of the actual step taken (or an estimate thereof).
Let's suggest two alternate hypotheses, A1
and A2
that cover all cases, and we'll let B
be our evidence (either a logical proof or the presence of an argument from authority or whatever). Then, according to Bayes' theorem
p(A1|B) = 1/(1 + (p(B|A2)p(A2))/(p(B|A1)p(A1)))
or equivalently, where !
means "not",
p(A|B) = 1/(1 + (p(B|!A)(1-p(A)))/(p(B|A)p(A)))
Let's try an example to see what these things mean. Let's let A
be our thesis, and B
is "an expert says A
is true". p(A|B)
is our estimate of how likely our thesis is presuming that an expert says it's true.
Let's evaluate each of the terms on the right.
p(A)
This is how likely we think the thesis is in the absence of expert opinion. Let's suppose that we really doubt this claim, and are relying heavily on our experts, so p(A) = 0.001
.
p(B|A)
This is how likely it is that an expert will say a thesis is true if it is in fact true. Assuming we limit ourselves to considering experts who have an opinion on a thesis, this could be pretty high--let's say it's 99%.
p(B|!A)
This is how likely it is that an expert will say a thesis is wrong even though it's true (and again we'll restrict ourselves to experts who ought to know). Let's say they're pretty accurate, but someone will slip up sooner or later, so we'll say it's 5%.
Now, our radically unlikely hypothesis looks like so:
p(A|B) = 1/(1 + 0.05*0.999/(0.99*0.001)) = 1/(1+50.45) = 1.9%
Well. That's better,but it's hardly a proof.
If it had been a proof, we would have had p(A|B) = 100%
.
And the difference in these percentages gives us a quantitative measure of just how bad it was to use argument from authority instead of a proper proof.