We do backtesting in finance; that is, we guess hypotheses/premises, and then use previous data to verify it.
Is backtesting inductive or deductive reasoning? (Here we have hypotheses/premises first, so I am confused).
I have seen a lot of people seek observations/data in order to fit the hypothesis or to prove the premises which sounds like backtesting). Is this unscientific?
You should know two things:
These two imply that if you want absolute certainty then you ought to rely on only FOL reasoning. But if you don't mind some uncertainty, then you may want to use heuristics to guide some of your decisions. Heuristics often cannot be proven to give the right answer, but they seem to work well most of the time. Statistics is a branch of mathematics, based on theorems that are deducible in the basic FOL-based systems that empirically has never failed to give correct conclusions. But there are many ways to act based on statistical measurements, and all of these ways are heuristic!
Statistical testing cannot tell you whether a hypothesis X is true or not. It can only tell you that, if you can truly set up the exact same experiment many times and its outcome is truly random with distribution satisfying X (plus some mild conditions), then running that same experiment many many times yields a distribution that eventually stays close to a certain distribution Y, where Y is something you can deduce given X. The problem is, you only do a finite number of experiments, not infinitely many, so even if you observe their results and compare against Y, it doesn't allow you to deduce anything about whether X is true or not!
You can use a heuristic of using 95% CIs (confidence interval), but this necessarily implies that you must accept that if you use this heuristic 20 times you cannot expect to make the correct conclusion more than 19 times on average! You can of course reduce the likelihood of making wrong conclusions by using 99% CIs, but no matter what you can never tell when you are making wrong conclusions, even if you only used the heuristic once! In case it's not obvious, there are no 100% CIs; those would be achievable only via FOL reasoning, not via statistical methods.
Back-testing is likewise a class of heuristics. One useful subclass is cross-validation. If you have good reason to believe that you can split your training data into k independent pieces, then you can apply cross-validation to those pieces and get k 'experiments'. If your hypothesis has good performance on those k pieces on average, then you could (heuristically) guess that it would do well on unseen but similar data too, and as k increases you can have greater confidence that your back-testing is a reliable indicator of performance on other similar data. But if those k pieces could be dependent, then increasing k doesn't increase your confidence! And if future data may be drastically different from your training data, then your confidence could be misplaced! And of course, being a heuristic, it may simply be wrong by sheer luck, because you literally only have one fixed set of training data.
Backtesting would generally be considered a form of inductive reasoning rather than deductive reasoning. Deductive reasoning involves drawing conclusions that are guaranteed to logically follow from stated premises. For example:
Premise 1: All men are mortal. Premise 2: Socrates is a man. Conclusion: Therefore, Socrates is mortal.
The conclusion necessarily follows from the premises based on the rules of logic.
Inductive reasoning involves making generalizations from specific observations. Conclusions reached via inductive reasoning are probable but not certain. For example:
Observation: The sun has risen every day for as long as we have recorded. Conclusion: The sun will probably rise tomorrow.
The conclusion is likely given the evidence, but not guaranteed. Tomorrow the sun could fail to rise.
Backtesting fits inductive reasoning because it draws probable conclusions about the future from limited past data:
However, this conclusion is not guaranteed or certain - the future may differ from the past. So backtesting supports inductive conclusions, unlike deductive reasoning which gives definitive conclusions. The premises alone do not guarantee the strategy will work, we need to gather evidence from data.
