Trouble understanding how the Chinese room experiment refutes computer functionalism

Question

I'm listening to John Searle's lectures on the philosophy of mind (https://www.youtube.com/watch?list=PL039MUyjHR1wfJpULVP1a1ZeCBmIHmhxt) and I don't really understand the significance of his Chinese room thought experiment. To me it seems to boil down to "imagine a program that only deals with syntax, then that program can't deal with semantics", but that doesn't show that no program can deal with semantics. Is there something I'm missing?

Answer 1

This is one of the most-discussed arguments in the philosophy of mind. The discussion encompasses almost every important topic in the discipline. As such, I won't touch on its implications and influence on the field.

---

First let's start with putting the actual argument (for an easy summary, taken from Wikipedia):

suppose that artificial intelligence research has succeeded in constructing a computer that behaves as if it understands Chinese. It takes Chinese characters as input and, by following the instructions of a computer program, produces other Chinese characters, which it presents as output...

Searle then supposes that he is in a closed room and has a book with an English version of the computer program, along with sufficient papers, pencils, erasers, and filing cabinets. Searle could receive Chinese characters through a slot in the door, process them according to the program's instructions, and produce Chinese characters as output...

Searle asserts that there is no essential difference between the roles of the computer and himself in the experiment. Each simply follows a program, step-by-step, producing a behavior which is then interpreted by the user as demonstrating intelligent conversation. However, Searle himself would not be able to understand the conversation. ("I don't speak a word of Chinese," he points out.) Therefore, he argues, it follows that the computer would not be able to understand the conversation either.

Essentially the root of the argument for Searle is the concept of "intentionality" (from SEP):

intentionality is the power of minds and mental states to be about, to represent, or to stand for, things, properties and states of affairs. To say of an individual’s mental states that they have intentionality is to say that they are mental representations or that they have contents.

What Searle claims is, given the fact that I can simply replace the AI, a black-box (where I cannot possibly say if it has intentionality or not) with a white-box situation where I can prove to not have intentionality - I proved that the AI does not have intentionality. This helps Searle argue against what he calls "Strong AI", i.e. that view that AI does not merely simulates the brain but is actually exactly the same.

---

This is where it starts to get a bit tricky, because, like every philosophical idea, the discussion of refutations begins. But I'll attempt to summarize the majority of the refutations to simply one point:

At the heart of Searle's argument lies the assumption that the white-box situation must be taken apart (a sort of reduction of the situation), when considering the man in the situation to not understand Chinese. But, we can easily argue that this assumption is wrong and instead consider taking the situation as a whole (i.e. the entire room operates as one organism/machine, sort of holistic view of the situation). This way, we can argue that the room itself understands Chinese, even if by way of reduction the man inside does not. This touch a far wider issue in philosophy as a whole and particularly philosophy of mind - the problem of holism versus reductionism (which is most prominent in the discussion of Emegentism, beautifully illustrated in Hofstadter's Godel, Escher, Bach).

---

Now of course there are many replies and piles of refutations and rejections of them, and this isn't the place to discuss all of them. I would suggest however, if you want a fuller picture, to read the SEP article on the topic. Also Partially Examined Life has a nice few episodes regarding functionalism (221-223).

Answer 2

but that doesn't show that no program can deal with semantics.

It does in a way if you accept what Searle's definition of a program means. He means, what computer engineers mean when talking about programs, a set of instructions.

The significance of chinese room argument(CRA) is to show that artificial intelligence can never have true understanding, at least in the intuitive sense of the word, but the way understanding is defined is itself ambiguous. This is also seen in modern AI image recognition systems which perform object recognition from pictures, and pictures that require you to recognize the background and put the object into context, often trip the AI and give wrong results. This is not because the AI failed to classify the geometry of the object by analyzing where edges begin and how colors change, but what the background is about(whether its a sunny beach or a parking lot). This could be an indication that we would need to build more and more complex AI that can recognize the entire thing, not just objects in the picture, but the fact that it is "seeing" a picture.

With that said, let us get a bit deeper into the analysis. CRA is basically a human acting out instructions that he reads in the rule book, but this supposes that the human "understands" the English in the book. From the outside, the room understands Chinese and that's good enough for a native Chinese speaker, but from the inside, the same cannot be said about any individual component (the walls, the table, the book or the human operator). It is our intuition that we understand things, however the point of this experiment is to show that this "intuition" cannot be explained by studying the individual microexplosions ongoing in the neural synapses of a human being. When the human reads English, what is actually going on inside him? Energy is being shuffled by motion, and it's simply a rearrangement of particles that take the human from a non-understanding state to an understanding state as seen from the outside. From the human's perspective however, the "feeling" of understanding is very different. This intuitive feeling, cannot be accounted for in a mechanistic description of the system alone, leading Searle to state that any computer system cannot have true understanding or at least in the way that humans feel it to be so, but there's absolutely nothing preventing this in principle. If we follow this argument to ourselves, then we too don't have true understanding of anything at all.

Answer 3

This may seem simplistic, it is not: It boils down to this:
No matter how well a machine can simulate a human being with a mind it is still essentially only gears and pulley's on the inside.

This continues to remain true no matter how good the simulation becomes. Even if the simulation becomes so good that there is literally no discernible difference what-so-ever between a living human mind and the Chinese room machine it is still only gears and pulleys on the inside.

At some point after it is fully self programming and can simulate free will, empathy, personal insight, a human personality with individual personal preferences and attitudes we could call it the functional equivalent of a human mind. At this point it may seem reasonable to grant it human rights.

Since the original post has a bad link here is a good link
The Chinese Room Argument
https://www.youtube.com/watch?v=18SXA-G2peY

Answer 4

"but that doesn't show that no program can deal with semantics."
Yes you are correct. I had to watch this video before I could understand your basis:
https://www.youtube.com/watch?v=18SXA-G2peY

In order for the Chinese room machine to perfectly simulate a human mind it must also perfectly simulate actual comprehension of the complete semantics of its input.

The semantic rules are specified and processed syntactically by forming the same relations between finite strings that a human mind would form between words. (The Cyc project does it this way using its CycL language).

Because the Chinese room produces the functional equivalent of human comprehension we can say that it does demonstrate the comprehension of semantics.

At the point in time that fully it demonstrates all of the human characteristics including: free will, personal preference, psychological attitudes and has a fully functional personality we might just give it the benefit of the doubt and grant it human status.

Answer 5

I think Searle's argument is still more powerful than is acknowledged here. To see why, consider the definition of a Turing machine: https://en.wikipedia.org/wiki/Turing_machine I won't go over the definition here, but suffice it to say that this is rote symbol manipulation par excellence. And yet it is widely believed (Church-Turing thesis) that that simple model captures anything that can be done computationally, including any artificial intelligence as we commonly think of it. Certainly it captures anything that today's AI is doing.

Now, as pointed out, there is an issue of scale. If I think about a simple Turing machine, I can work out its steps one at a time. But if I think about sophisticated AI, that's no longer very helpful -- it's just too much for me to try to simulate all the steps in my head. We don't have to think about human-level AI for that; even if I play Go against AlphaGo, it is more practical and effective to think of it as understanding the game and trying to beat me (and surely succeeding at that). So purely for practical purposes it makes sense to think of it as understanding the game.

But there are also senses of the word "understanding" for which we might be skeptical that AlphaGo understands Go. Is there actually something it is like to be AlphaGo? Does it have qualia? Does it have a true appreciation of what it is actually doing when it is playing Go? Many, including most AI researchers, would say that it does not, because we still know that at bottom all that is going on is lots of tiny steps of computation being followed in a rote fashion, possibly distributed across multiple machines, and there seems to be no reason to think that there is any centralized higher-level "awareness" of the game anywhere.

But now suppose we actually succeed at building human-level AI, and assume we do so in a way that is not drastically different from how think about computation today -- and this is the point of the Chinese room, that the way we think about computation today, anything we do could be implemented by a Chinese room (albeit extremely slowly), because it simulates a Turing machine. Then, if we're skeptical about AlphaGo really "understanding" Go, then why shouldn't we be just as skeptical about this human-level AI having any real understanding of whatever it's talking about? When I understand something, there is something that that is like, there is a sort of apparently centralized awareness of what is being understood. But it is not clear why we should think of the human-level AI as having such apparently centralized awareness, if we just examine the mechanics of how it works, for the same reason that we may think that AlphaGo doesn't have it.

And yet of course there is a strong intuition as well that if we interrogate it and it answers like a human being would, we would think it's really understanding. So I'm not saying that I completely buy Searle's argument. But I think it raises a very difficult question and it hasn't been decisively refuted. Of course we similarly still really don't understand how all our own neurons firing somehow creates the kind of awareness we have. That is kind of the point of the argument in my view -- it's raising some hard questions, pointing out that there are still real gaps in our understanding of these things, and as you can probably tell from the above, I think they're closely tied to some of the hard problems of consciousness.

Answer 6

Searle’s argument has been rejected or refuted - whatever you prefer - in Hofstaedter’s famous book about Gödel, Escher and Bach.

He is basically bamboozling you by mixing up scales. Your brain is made up by a few billion brain cells. These brain cells do all the data processing. None of the brain cells has any understanding, or any consciousness. Yet you have consciousness and you understand things.

He maps the brain to a room, not filled with billions of oversized brain cells, but with one single human manipulating symbols. That human doesn’t understand any of the symbols. Therefore, says Searle, the room doesn’t understand anything.

But he is playing a trick. By having one human only doing the symbol manipulation slowly instead of a billion cells doing it much faster, he creates a speed difference of at least 1 : 10 billion, likely more. So what you do in a second, takes this poor man at least hundreds of years. Our mind wouldn’t attribute understanding and intelligence to a system that takes 300 years to react. But make it ten billion Times faster. Now you have something that reacts similar to a human. You can have a conversation with it. And at that point there is no good reason to say that this room has any less understanding than you do.

And we still haven’t developed a program that would have that kind of understanding, so right now ther is no “understanding” computer. That doesn’t mean it cannot be created.