Big Data's effects on questions in philosophy of science

Question

The Oxford English Dictionary recently named ‘post- truth’ its word of the year. The term, whose use is reputed to have increased 2000% in the past year, is defined as: “…relating to or denoting circumstances in which objective facts are less influential in shaping public opinion than are appeals to emotion and personal belief.”

Which seems to refer to the ostensible prevalence of fake/false facts now so frequently promulgated in social media. On hearing the definition, I immediately transposed "belief" and "opinion" to create a sort of feedback loop and [further or alternatively] defined the term as “circumstances in which objective facts are less influential in shaping personal beliefs than are appeals to emotion and public opinion.”

This formulation brought to mind a question I posed here a year ago (but which, despite very good answers from Messrs. King and Alexander, was soon closed as too broad), relating to another way that computers (here in terms of computational capacity rather than possible alternative sources of/or authority] have impacted our relationship to “facts,” both of which contribute to the kind of fact fatigue that many of us seem to be experiencing.

My new question (the old rephrased and expanded upon) is: How will the increasing prevalence of “big data” – the exploding plethora of information and computing power to correlate it – impact:

(i) The scientific method’s theory/hypothesis formation/confirmation/falsification; defining theory/hypothesis broadly as something like a tenable, provisional assumption made in order to extract/determine its empirical (and maybe, depending upon the relevant fach, also normative and/or logical) consequences;

(ii) The exposition of Quine’ the underdetermination thesis; i.e. how will the phenomena affect the perception that theories are underdetermined because at any given time conflicting theories are consistent with the [ever increasing] data on hand; and

(iii) The realist/relativist/ constructivist debate? For instance, as Nelson Alexander observed, “Since very little science can now be done without the "sensory apparatus" of massive data crunching [likening it to Galileo's telescope], we seem to be slipping unavoidably into a more "constructivist" circle of confirming-predicting.” Are there also conceivable ways in which the phenomena of big data can be said to support a realist ontology/epistemology?

While this “Question” constitutes three queries, I believe that considering them together will increase the relevance and quality of the answers, because all three legs deal with how observables (consequences/data, however one defines them/it, [can be said to] “explain”, “construct” [or be correlated with] non--observables.

Answer 1

Preliminary point 1: There are a handful of philosophers of science whose current work is focused on big data. I especially recommend Luciano Florid and Sabina Leonelli.

Preliminary point 2: Philosophers of science dislike talk about "the" scientific method. Particle physicists, climate modelers, and ethologists, for example, take very different approaches to designing research projects, gathering data, and analyzing it. Things get even more diverse when you include qualitative social scientists and historians.

(i) In 2008, Chris Anderson, the editor in chief of Wired, argued that big data was bringing about "the end of theory"; that working with big data was theory-free, non-causal, inductive science. I think some people have compared this to Bacon's inductivism, in contrast with Popperian falsificationism (though I can't find any references at the moment). Anderson's is the most radical position I've seen regarding question (i). Ratti and Leonelli have both argued that big data is not novel, or at least not novel in the way Anderson presents it. (See also this review/commentary paper by Mazzocchi.) If I recall the papers correctly, both Ratti and Leonelli argue that Anderson's theory-free, non-causal, inductive science is exploratory research, and that science is very familiar with both exploratory and confirmatory modes of research. Leonelli also points out that some traditional small data statistical concerns — such as sampling error — are still highly relevant to big data.

(ii) I have two answers here, one for the process of gathering and maintaining big data, and one for the process of analyzing big data to build models and draw inferences. First, big data gathering is just as hard, messy, and contingent as small data gathering. For example, gene sequencing involves cutting the DNA of interest into many, many short segments, chemically replicating these short segments so that you have enough to read reliably, and then stitching the results back together by trying to match up overlapping sequences in the short segments. As I understand it, there are a few different methods for stitching the results, and they don't necessarily agree. There are also prior decisions about where to gather data from. For example, there have been multiple analyses showing that genomics research tends to focus on people of European ancestry. And there are important contingencies about what metadata are attached to a given piece of data (should we be attaching race and ethnicity metadata at all?) and which data are kept or discarded. In all of these cases, different paths could have produced substantively different data, and so there's a sense in which the data themselves are underdetermined.

Second, given a data set and research question, there are many possible ways to analyze the data and attempt to answer the question. In the small data realm, this gives rise to the possibility of p-hacking and the garden of forking paths: basically, if you keep trying different analysis approaches, eventually you'll get a "statistically significant" result. (Compare with this critical review.) I would argue that big data enables more p-hacking than small data, in three ways. First, big data involves much larger samples, which increases the statistical power. That might seem like a good thing, but it means that traditional statistical methods can become overpowered: even the smallest differences count as "statistically significant." Second, big data often involves many more features (covariates, predictors, independent variables) than observations, such as thousands of genes for a few hundred individuals. This makes it extremely easy to overfit the data. Third, modern machine learning approaches often require big data. But traditional statistical methods can still be applied to the same big data sets. So big data analysts have more possibilities — more forking paths to follow, more ways to hack out some statistically significant p-values — than small data analysts.

(iii) I don't know who Nelson Alexander is, and can't find any context for the quotation online, so I won't comment on that. Anderson's view of big data (see the answer to question (i)) lends itself to a purely instrumentalist account: we're finding correlations or other statistical patterns in the data, not discovering causal relationships. On the other hand, in the context of the replication crisis, at least three different people have suggested a connection between replication problems and lack of good causal theories that can inform and data gathering, analysis, and interpretation. In other words, the argument goes, without causal theories, you can't design your experiments, observations, and data analysis in a way that control for potential confounders; and so you end up with replication problems. This could be seen as an argument against instrumentalism. Many modern machine learning approaches (random forests, support vector machines, deep neural nets) are best understood as purely instrumentalist; but other approaches developed by AI researchers, such as Bayes or causal nets, aim at causal reasoning, and so seem to fit better with a realist view. (Personally I tend to view things through the lens of model-based science, which is orthogonal to the realism/antirealism debates; for instance, I like this paper by Potochnik.)