Is it possible to have a scientific review of a method if the author doesn't have direct experience of it?

Question

It seems like it is possible to conduct scientific research without actually getting close to the sample/specimen. However, what if the "sample" of the research is a method? For example, there are many methods to dissect a whale. Is it possible to have a scientific review of a method if the author hasn't had direct experience on that method, but only know its rules or steps, and other researches talking about it? And if the answer is yes, then how is they relatively confident that they don't miss something (the unknown unknowns)?

I suppose there are different cases to consider:

• The author doesn't have direct experience on whales
• The author has direct experience to whales, but doesn't have direct experience on any method to dissect a whale
• The author has direct experience to whales and methods to dissect a whale

_{According to this meta post, this question invites answers to dissect what the question means. Such analysis on possible interpretations of the question can constitute an interesting philosophical analysis.}

Answer 1

This question misses a key point. One doesn't do research on things — in some nebulous, abstract sense — one does research to understand properties of things. Newton was not interested in objects with mass in and of and as themselves. Newton was interested in things like inertia and acceleration under gravity: how objects of mass behave under different conditions and contexts.

One would not do research into whale dissection in itself; what would be the point of that? One could do research into which whale dissection method was most efficient, least costly, speediest, least wasteful, or other measurable effects that a given method might show. One can make perfectly valid inferences based on the rules and steps of the method — e.g., I infer that any whale dissection method that prioritizes the use of butter knives will score low on most metrics (except perhaps the 'style' metric) — but ultimately we have to have data on the various methods that we can analyze. Does this mean we personally need to go watch whales be dissected? Not necessarily: I mean, someone has to go a-whalin', somewhere, but we can collect and analyze data from them, without even having read Moby Dick.

Answer 2

I laid out the argument that science is what scientists do here Philosophical assumptions underlying science

One-off or near unique events can still be investigated by science. Chixulub, gamma ray bursts, the big bang - we study that and are as 'far' from it as it's pissuble to be, in space & time. There was a first ever dissection of a whale done with science in mind. A less experienced scientist isn't necessarily less scientific - in fact almost every Nobel Prize was received for work done before age 30.

if the answer is yes, then how is they relatively confident that they don't miss something (the unknown unknowns)?

We develop domain-specific toolboxes. Consilience or convergence of evidence is a good start. Correcting for cognitive biases, with tools like double-blind trials, might be needed. Galen caused over a millennia of medics to ignore their own eyes during dissections, because he had nit dissected humans, only pigs & chimpanzees &c.

I think of the detection of gravity waves as a masterclass in how science is done. Each source of noise was systematically minimised, given the budget, with steps like damping, multiple separated detectors to eliminate noise, and error bars chased to below the signal range (though in fact, thus scale was unknown).

In the whale example, there are clearly transferable skills from dissection & other surgical procedures. Literature may indicate key whale-specific issues - for rabbits I think the gall bladder should be removed quickly or it 'taints' the meat, which might be relevant to investigations. Sperm whales alone produce ambergris, which might warrant specific investigation in that case. Whale-strandings have focused on ear damage, which has required developing more detailed knowledge of the minutiae of dead healthy whales - it's thought military radar has been causing deafening sound bursts, resulting in certain microscopic air bubbles, causing pods to flee the water. There's always going to be the cutting-edge of research doing new techniques, and work-a-day routine science - but the boundaries of both continuously expand.

There's a whole deeper level about hypothesis generation & falsifiability, creating tractable abstract models, scientific paradigms, but that doesn't seem like what you are asking. It seems like you are trying to get at something deeper that you haven't made explicit..

I think the summary is, being better able to review methodologies, and think through unknown unknowns is what we call being a better scientist. Often before the science is done, we don't know who is good, so we use proxies like exams & interviews. But the good science of better doing these things, results in noteworthy insightful science, and we try to elevate those who can do that to supervise & teach.

Answer 3

Yes. Science is done on datasets and theoretical models,¹ not necessarily on direct experience:

      Underground nuclear test

 

      Neutron star

I can only add that in general, we don't need to experience something directly in order to feel it or live through it. We think it by daydreaming it. After doing it for your whole life, it gets pretty real. That's our superpower as humans -- sharing our individual experiences with each other.

¹ except, that nowadays, theoretical models get routinely neglected in favor of the data science resulting in a frightening number of peer-reviewed studies doing something like this: