So I was watching this and was wondering about the consequences and importance of the "No Free Lunch Theorem"

David Wolpert: that if I have any particular reasoning algorithm be it in say the machine learning algorithms or optimization algorithms evolution itself any of which is a search algorithm and if I find that of my two algorithms a and b that for one set of priors a outperforms b also it's going to make more accurate predictions about whether the sun will rise tomorrow or what have you that foreign for there is always going to be just as many priors for which b beats a


Kurt: what I mean is people including myself perhaps I'm mainly speaking about myself have regret over the past and squandered opportunities feeling like feeling like I didn't implement the optimal strategy I haven't lived up to my potential now is that a valid concern or is your theorem nihilistic where it doesn't matter what you do?

I suspect the only way out is intuition? Our intuition is a product of evolution in our environment. This bias introduces a bias in the "search algorithm" one is likely to use in life. Are there any other way-outs to Kurt's conundrum?

1 Answer 1


The No Free Lunch theorem states that when averaged across all possible problems, any two strategies have equivalent performance. At first glace, this might suggest it doesn't matter what strategy you use for any problem, but that's not really the case - there's a wide array of literature detailing the successful application of machine learning methods to particular problems where some methods reliably outperform others. The key is to recognize that very rarely are we trying to solve all possible problems. Problems typically encountered in day-to-day life make up a small subset of all problems. No free lunch applies when you can make absolutely no assumptions about the data or target functions, but we virtually always have baseline priors of what to expect when trying to solve a problem that's similar to ones that have been solved before - as the OP puts it, "intuition".

For particular problems, there certainly are some algorithms that are better than others. It's not possible to pick a strategy that's optimal in all circumstances, but it is possible to pick better or worse performing strategies for particular problems. Any situation in life represents a particular problem, for which you cannot take the nihilistic view that the strategy doesn't matter. If you want to solve a particular problem like classifying images of cats and dogs, for example, a random classifier will likely not perform as well as an image recognition approach. No free lunch states that if you applied those same algorithms across every possible input/output space, neither would perform better than the other, but the specific task of telling cats from dogs does not occupy the entire space of all possible inputs/outputs.

  • So kant's innate knowledge as been vindicated? Assuming the answer to " Are there any other way-outs to Kurt's conundrum?" is no? Commented Feb 3, 2022 at 17:02
  • No, learning skills about how to make decisions in our world is not just "intuition". There are life skills that apply across multiple problems, as much of our world has similar features that make those skills transferrable problem to problem. Wisdom can be learned.
    – Dcleve
    Commented Feb 3, 2022 at 17:27
  • @Dcleve feel free to write an answer which elaborates your point? Also are you saying no to "So kant's innate knowledge as been vindicated? " or " Are there any other way-outs to Kurt's conundrum?" Commented Feb 3, 2022 at 17:34
  • The reason I say intuition is because when I was born I didn't solve a single problem and yet here I am :) Commented Feb 3, 2022 at 17:40
  • @MoreAnonymous -- Unconscious learning of more successful decision strategies is part of the process of developing wisdom. Guiding that process consciously is generally highly beneficial. See Thinking Fast and Slow for a good discussion of how we develop unconscious skills, but need to check and steer those consciously.
    – Dcleve
    Commented Feb 3, 2022 at 17:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .