Consequences and importance of the "No Free Lunch Theorem"?

Question

So I was watching this and was wondering about the consequences and importance of the "No Free Lunch Theorem"

David Wolpert: that if I have any particular reasoning algorithm be it in say the machine learning algorithms or optimization algorithms evolution itself any of which is a search algorithm and if I find that of my two algorithms a and b that for one set of priors a outperforms b also it's going to make more accurate predictions about whether the sun will rise tomorrow or what have you that foreign for there is always going to be just as many priors for which b beats a

...

Kurt: what I mean is people including myself perhaps I'm mainly speaking about myself have regret over the past and squandered opportunities feeling like feeling like I didn't implement the optimal strategy I haven't lived up to my potential now is that a valid concern or is your theorem nihilistic where it doesn't matter what you do?

I suspect the only way out is intuition? Our intuition is a product of evolution in our environment. This bias introduces a bias in the "search algorithm" one is likely to use in life. Are there any other way-outs to Kurt's conundrum?

Answer 1

The No Free Lunch theorem states that when averaged across all possible problems, any two strategies have equivalent performance. At first glace, this might suggest it doesn't matter what strategy you use for any problem, but that's not really the case - there's a wide array of literature detailing the successful application of machine learning methods to particular problems where some methods reliably outperform others. The key is to recognize that very rarely are we trying to solve all possible problems. Problems typically encountered in day-to-day life make up a small subset of all problems. No free lunch applies when you can make absolutely no assumptions about the data or target functions, but we virtually always have baseline priors of what to expect when trying to solve a problem that's similar to ones that have been solved before - as the OP puts it, "intuition".

For particular problems, there certainly are some algorithms that are better than others. It's not possible to pick a strategy that's optimal in all circumstances, but it is possible to pick better or worse performing strategies for particular problems. Any situation in life represents a particular problem, for which you cannot take the nihilistic view that the strategy doesn't matter. If you want to solve a particular problem like classifying images of cats and dogs, for example, a random classifier will likely not perform as well as an image recognition approach. No free lunch states that if you applied those same algorithms across every possible input/output space, neither would perform better than the other, but the specific task of telling cats from dogs does not occupy the entire space of all possible inputs/outputs.