According to PhilPapers Survey from 2020 there are 239 philosophers who identify as accepting or leaning towards Physicalism and yet reject consciousness of future AI.
To me this seems very puzzling, what is the difference between brains and future AI under physicalism?
Technically speaking, the question can only be answered speculatively since the motivations of a group of philosophers is best answered by empirical means, that is, polling. But I'm on lunch, so I'll speculate as a follow-up to a full-throated defense of AGI as an example of empirical philosophy (PhilPapers). As both a moderate functionalist (SEP) and a physicalist, one might observe some following facts about philosophy and philosophers.
Let us observe:
P1. All known philosophers are human.
P2. All humans have emotional states and logical shortcomings.
C. Therefore, all philosophers have emotional states and logical shortcomings.
From a naturalized epistemology, the basis can be found in behavioral economics and evolutionary psychology, so we won't dwell, but the fact is that human beings aren't entirely rational creatures as traditional philosophers of economics have us believe by way of their Homo economicus claims. In fact, being very rational is a life-long pursuit, and few even aspire to it, and clearly there are psychological biases to fearing machines that approximate humanity, most famously the uncanny valley.
In aesthetics, the uncanny valley... is a hypothesized relation between an object's degree of resemblance to a human being and the emotional response to the object. The concept suggests that humanoid objects that imperfectly resemble actual human beings provoke uncanny or strangely familiar feelings of uneasiness and revulsion in observers.
Already, there is much philosophical discussion, including an ethical dimension about conventional AI, such as Kate Crawford's Atlas of AI. Three million people have trucking-driving related jobs in the US alone, and AI is on the verge of being safe to use to eliminate people from those jobs. History is full of people whose income is eliminated by new technologies protesting and even destroying equipment. As the Sinclair once remarked, “It is difficult to get a man to understand something, when his salary depends on his not understanding it”. And that's not just literal salary. Take a professional philosopher like John Searle whose Chinese Room is an important argumentation because of his academic reputation. If a computer beat him in a philosophical debate, it seems unlikely he'd still deny computers can think.
As many philosophers believe, truth can be constructed by society. Echoes of these claims happen in the philosophy of science with Kuhnian thought, in the social sciences with social constructionism, and are hallmarks in historical debates of philosophy and science so much so that Max Planck once quipped:
A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die and a new generation grows up that is familiar with it... An important scientific innovation rarely makes its way by gradually winning over and converting its opponents: it rarely happens that Saul becomes Paul. What does happen is that its opponents gradually die out, and that the growing generation is familiarized with the ideas from the beginning: another instance of the fact that the future lies with the youth.
And it's not just contemporary philosophers, but theologians too. Giordano Bruno is a famous example of a philosopher who bucked the dominant cosmogony and paid for it with his life partially because he challenged geocentrism.
So, now to respond to your question in more detail, why would physicalists reject the possibility of AGI?
So, just because someone is a philosopher and espouses a physicalist position doesn't mean they have any real expertise in the philosophy of AI or know much about AGI. Honestly, most people who espouse opinions on AGI couldn't even tell you what an unsupervised learning model is, let alone describe how it works or tackle the philosophical implications of connectionism.
It is perplexing, isn't it. I've encountered it on here, eg in the comments to this: Can minds be uploaded in computers? which is also an answer relevant to this topic, about the challenges of simulating embodied and extended cognition and how even quite a long horizon of AGI might not be able to simulate minds much like our own.
My sense is that physicalists who full-stop reject synthetic sentiences, or artificial general intelligence, object to it in two types of ways: structure, and process.
Roger Penrose is an example of a physicalist rejecting AGI being possible on Turing Machines, which he covers in The Emperor's New Mind. He thinks Godel Incompleteness means something fundamentally different is required, and looks to a process of 'orchestrated objective reduction' as concentrating information. That may be a quantum process, but there are researchers working on it being a classical process.
Penrose and others who object to AGI in process terms, whether quantum or not, seem to be missing the fundamental universality of Turing Machines, including quantum ones. Maybe an exception, is Deutsch & Marletto's development of Universal Constructors, which originated with VonNeumann's extension of Universal Turing Machines to counterfactuals, which allows computational intensity and the Halting Problem to be dealt with in very natural ways.
The summary of the structure arguments seems to be, we don't yet know some special quality about how human brains function. Rather than it being impossible to simulate in principle.
I would call raising the issue of qualia, an example of a process objection. Something about how a specific mind in a particular body, leads to qualities of experience related not only to present circumstances, but to the accrual of them in a way that cannot be abstracted or separated from the history of having been that individual.
Chalmers Fading and Dancing Qualia thought experiments take aim at this objection, in principle. A lot of ink gets spilled over qualia, for few clear words though. It seems there is a culture-gap issue between those from a computation & information theory background, and professional philosophers, that tends undermine productive discussion. There are so many unknowns, those two broad tribes can retreat into their silos if they want. I like people like Anil Seth, who take a middle path, of acknowledging the scale and uncertainties of the challenge, but also that we can genuinely make headway in deepening what we know.
For me a deeper process objection, is how animal minds seem to need this tightrope between positive and negative experiences, and that going too far either way leads yo stagnation, or suicide. Species have evolved to manage. And individuals have carers and community to help them manage the tensions, as their capacities increase. Eg boredom is very unpleasant for children, while they are in peak learning years. It seems a substantial challenge might be involved in cultivating a synthetic intelligence, perhaps requiring some kind of ecology, and interacting evolutionary algorithms. Crucially though, any progress on computer minds can be saved and copied, being substrate-independent.
I would look to the sense-gates model of consciousness found in Buddhist thought, to understand minds as arising through interaction, rather than simply being built on a plan. Our biology is full of cases of stem cells forming the correct tissue type in response to stimuli, rather than as directed by DNA. Our neural tissues ruthlessly prune unused pathways, and adapt pathways to deal with stimuli.
It's surprising to see a figure as high as 45% of this group of physicalists rejecting conscious AGI. I guess maybe the qualia objections are the most common, but as per Chalmers it's hard to see how anyone can expect that to be a fundamental problem.
From a mathematical point of view, by having a random generator we can make any types of distributions. A decision machine like AI, that works based on random generators and feedbacks, should theoretically simulate consciousness in every aspect. So, my guess is that lack of understanding about the math behind AIs could be the reason.
Random generator is a mathematical process to create random numbers so that when an AI don't know what to decide, decide randomly.
Feedbacking mechanism changes the random distribution to the correct answer distribution.
So, by having enough time to experience, an AI can be more intelligent and conscious than conventional human brains.
Assuming the truth of physicalism, we still have no idea how consciousness arises in biological systems. Lots of physicalists in the AI community assume it has something to do with algorithms or information or some other abstract property that can be emulated in silicon, but maybe it's not something abstract; maybe it's biology. Maybe it's something in living cells that gives rise to consciousness. Or maybe it's something that arises out of DNA. We really have no idea.
Why would you expect that simply simulating the behavior of a brain would create the side effects of the brain? Simulating the behavior of a dog with a robot doesn't automatically create the side affect of being able to produce puppies. A wax fruit is not good to eat. Why expect that the consciousness that arises out of a working human brain is different, that it will suddenly appear in any system that behaves like a human brain?
ADDED: In the comments, someone asks how consciousness evolved if it is not a consequence purely of the function of the mind. This is a more subtle problem than it may at first appear. First, note that we can't give an answer like "maybe consciousness itself had an evolutionary advantage" because in a physicalist setting, all behavior has to be ultimately caused by physics. Consciousness can't be causal on its own. At most it can be an epiphenomenon riding on top of the brain but having no causal powers. So it can't be directly functional.
One possibility is that the biological structures that give rise to human intelligence just naturally also give rise to consciousness. This would not imply that simulating, or even replicating, that behavior in silicon would also give rise to consciousness. For example, maybe the chemicals that get released into the blood, associated with feelings of fear, anger, lust, love, etc. act on the cellular structure of the brain to produce consciousness as a byproduct.
Another possibility is that the route that led to the human brain just happened to also lead to consciousness, just as, for example, the evolution of the eye was (probably) primarily directed at avoiding predators or detecting prey, yet as a side affect it also led to animals segregating into diurnal and nocturnal species.
Note that I'm not actually proposing either solution--they are entirely speculative; I'm just showing that the objection from natural selection has possible solutions.
Answering this question should NOT involve dismissing physicalists who reject strong AI as somehow confused.
Strong AI and the possibility that an AGI will be conscious, involves a set of assumptions that many other physicalists find suspect.
The key assumption is that consciousness by definition IS the execution of an algorithm. This is Algorithm Identity Theory.
There are multiple lines of objection to this.
The distinction between an algorithm, and the execution of an algorithm, is an intrinsically unclear distinction. If code is not conscious, and few AI theorists claim it is, then why would stepping through code line by line become conscious?
The Evolutionary Argument, developed by James relative to epiphenomenalism, and extended by Popper to all Identity Theories, notes the evolved nature of consciousness, and its only occasional and not "necessary" correlation with whatever it is "identified" with, combine to refute all Identity Theories.
Functional/Algorithmic Identity requires the algorithms be the substrate that consciousness be identified with, and this substrate has no mass, energy or location -- IE it is not physical. This is implicitly Kim's objection, when he rejects AIT in favor of Neural identity theory. Kim considers functionalism to be a form of dualism, and to break the causal closure of the physical.
Searle's objection is empirical and pragmatic -- we have 3/4 of a century of failed efforts to create consciousness using a variety of AIT assumptions for what triggers the "magic" of consciousness to emerge, and every one of them have failed for most of a century. Searle's inference is that the phenomenon that leads to consciousness emergence is biological, not functional.
Dennett and the delusionists assert that consciousness is a delusion created by the writing of memories into long term memory. why this would end up creating consciousness, he actually doesn't have an explanation, but the FUNCTIONAL aspects of intelligence and actuation, he holds do NOT involve consciousness at all. Hence per Dennett there is no reason to think that an AGI will bother to be conscious, as it is an irrelevant delusion.
The Churchlands and eliminative reductionists also think consciousness is a "delusion" in that it is an incorrect model that we have convinced ourselves of relative to our neural processing. Eliminativists think that with a better understanding of our neurology, the ideas behind consciousness will disappear.
With these many fatal objections, rather than surprise that as many as half of physicalists reject the consciousness of future AGI, instead that that ONLY half of physicalists reject the idea of an AGI being conscious, THAT should be the surprise here!!!!
Aside, as a spiritual dualist, I do not reject the consciousness of future AGI. I consider the conditions for ensoulment of a physical object, and causal interaction with spirit, to be very possible with future AI. However, I think most future AI will simply bypass that process, and not couple spirit into their causal process. But I consider the objections to algorithmic identity theory above to be very convincing, hence the plausibility of AGI consciousness within a physicalist worldview to be much more difficult, approaching impossible, to defend.
What is so surprising about this? The only notions of consciousness that we can reliably verify is biological. Even if you take the ardent skeptic route that we can’t verify others’ consciousness, we can verify ours. There is nothing that indicates that artificial intelligence, made of chips and transistors, can ever be conscious. Consciousness is a phenomenological quality that seems to be underpinned by biological processes
