I've got a huge dataset with hundreds of variables. However, I'm not sure which scientific method is more sophisticated in this case. The two scenarios are the following:

  1. Analyse my huge dataset first and formulate hypotheses later. I've got no preconception, and I try to come up with interpretation.

  2. Formulate the hypothesis first and analyse only the relevant part of the dataset later. It's crucial to have a testable prediction first, and I should have an answer whether the hypothesis true or false.

Regarding this topic, I had an argument with my (Economics) professor and a researcher. What's more, their opinions differed. I'm confused. Could you please help me to see this research question more clearly?

  • 1
    Hypotheses guide data analysis with minimal information based on the data. It is the theory that tells us what we can expect to see in the data. We have an intuitive idea that we commit to and then use the data to try to rationalize that intuition, however, we may have to revise that snap decision after the analysis. Justification for this process might be found in Jonathan Haidt's "The Righteous Mind". One might not be able to come up with any hypothesis of value by just analyzing the data. Feb 4 '18 at 18:27
  • 1
    Further to @frank-hubeny's excellent comment, the notion that there is a One True Scientific Method is a fiction that philosophers of science have spent decades critiquing and untangling; Chalmer's book "What is this thing called science?" provides a good overview. But on a lighter note, perhaps the professor and researcher might appreciate this comic on "The Scientific Method vs The Actual Method". :-)
    – Alexis
    Feb 5 '18 at 7:54
  • Since the topic is unknown here (viewed from the user's side) it is somewhat difficult to answer the question. But you say that the huge dataset has hundreds of variables. That means each one is significant. If you followed the second option, I believe, you would be away from scientific-method. So, I would prefer the first one. But IMHO, after analyzing the dataset you should classify the them very carefully into groups according to your requirements. Then you can formulate hypotheses. Feb 6 '18 at 15:46
  • You said it's crucial to have a testable prediction first. If so, you can make a testable prediction from a suitable group. I think you would be able to answer whether the hypothesis is true or false easily. Feb 6 '18 at 15:46
  • Doesn't it seem ironic that the choice could lead to different results and yet you still want to proceed under the pretense that you are following some sort of "scientific method"? It seems to me that if this were truly a "scientific" endeavor then either choice would lead conclusively to the exact same result.
    – Dunk
    Feb 6 '18 at 15:48

One common approach in statistics and certain experimental fields (like psychology) is to distinguish exploratory and confirmatory research. Exploratory research is open-ended; you can go in with some hunches or vague questions that shape what you'll look for, but you don't have a specific hypothesis. Confirmatory research starts with a specific hypothesis, and is designed to rigorously test that hypothesis. The two kinds of research involve different methods — exploratory data analysis for exploratory research; experimental design and statistical hypothesis testing for confirmatory research. This distinction is very similar to the old philosophy of science distinction between "the context of discovery" and "the context of justification", though I'm not sure whether there's a genealogical relationship.

Exploratory research is flexible and open-ended, but cannot claim to have "shown" or "proved" anything. That requires the rigor of formulating a specific hypothesis, designing an experiment and analysis plan, and only then collecting data. Using the same data to both develop a hypothesis and claim that you've confirmed it is sometimes called Hypothesizing After Results are Known, or HARKing (sorry for the paywall), and in the context of statistical hypothesis testing it produces incorrect inferences.

The exploratory/confirmatory distinction doesn't apply to all fields of scientific research; it only really fits experimental research, where in principle you can conduct carefully controlled, repeatable studies to collect more data.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.