5

I just can't see how John Searle's Chinese room makes sense. The room passes the Turing test. People outside the room think there's a human inside who understands Chinese. But, Searle explains, the room actually contains, in analogical form, all the essential elements of a electronic digital computer programmed to understand (according to Strong AI) written Chinese. But the monolingual English-speaking man in the room (the computer CPU) understands no Chinese. Cards inscribed with Chinese symbols fall into the room through a slot in the door. These are sensible Chinese questions. The rule book (the program) deals only with their shapes, not their meanings. It instructs the man to find certain Chinese characters among the spares in the room then to push them out through the slot. Unknown to the man, these are sensible Chinese answers. Neither the man nor the room understands the meanings of the shapes, since all they have is the shapes. From here Searle goes on to argue that computers will never understand language or the world.

What seems to me like a fundamental mistake is that Searle bases his argument on comparing a computer receiving Chinese symbols with a human receiving Chinese symbols. Then from the fact that the computer doesn't understand the meanings of the symbols, Searle argues that computers could never understand anything.

Well, humans can't understand the meanings of the symbols either. Humans first have to learn Chinese. Why doesn't the room try to learn Chinese? Without this, Searle's argument is pointless. Leaning Chinese entails developing memory structure. There's no structure in the Chinese room because there is nothing in the room to build it out of. The room's ontology needs structural elements added to it so that it then contains atoms of structure as well as symbols (the content of structure). Then the program can instruct the man to build memory structure. Digital computers can easily build memory structure and often do. Now with structural elements, the Chinese room can try to learn Chinese. And by the way, the CRA is unsound because Searle's premiss "... a digital computer is a syntactical machine. It manipulates symbols and does noting else" (John Searle, 2014, "What Your Computer Can't Know", in The New York Review of Books, October 9, 2014) is false. Also, it can be well argued that some structural elements are semantic.

It would be really great to get some comments and criticisms of my above reply to the Chinese room argument.

  • 3
    The rule book contains structure, doesn't it? – Quentin Ruyant Jan 19 '18 at 22:14
  • 4
    "Also, it can be well argued that some structural elements are semantic." -- I've often seen that argued; but never well argued. Name a single semantic element of a digital computer. – user4894 Jan 19 '18 at 22:47
  • You write: The rule book (the program) deals only with their shapes, not their meanings. However, that is all the human knows of the Chinese language. The program is assumed to be able to take in characters in a given order and return a meaningful response. The program has a process that allows it to do more with the characters than the human can. The point is that following that process will not generate understanding of Chinese since the human can follow the program, but not understand Chinese after doing so. Hence there is more to understanding Chinese than any program can contain. – Frank Hubeny Jan 20 '18 at 0:06
  • the room has lots of memory. think about the amount of information encoded in the look-up table, whose size would dwarf the known universe. or maybe this is your point. – Tim kinsella Jan 20 '18 at 2:18
  • 1
    @ user4894. "Name a single semantic element of a digital computer". For a start it must concern sensory symbols since these are the causal consequents of the external world. A sensory surface has detectors each of which reacts to a different small range of some aspect of the external world (eg sound frequency). When activated, a detector emits a unique symbol. The relation of temporal contiguity between the emitted symbols mirrors the temporal contiguity between the external events that caused the symbols. The temporal contiguity between symbols can be permanently recorded using pointers. – Roddus Jan 21 '18 at 3:08
5

Even if the man inside the Chinese room memorised every single translation instance (theoretically every possible combination which is impossible given our limited memory, but it's a thought experiment, so this constraint doesn't matter), would he understand Chinese, since he doesn't understand the meaning of any of the cards he has been presented with?

Searle does not think so:

in the literal sense the programmed computer understands what the car and the adding machine understand, namely, exactly nothing. The computer's understanding is not just (like my understanding of German) partial or incomplete; it is zero. . . . In the linguistic jargon, they have only a syntax but no semantics.

It is hard to say, therefore, that the man in the Chinese room does understand Chinese, as he is just following a set of rules or algorithms, much the same as an AI or super-intelligent computer that appears conscious would be doing.

What's central to this is the idea of human consciousness and whether it can be, firstly, defined and, secondly, simulated on a computer. If we don't know what human consciousnesses is, we have no chance of simulating it on a computer. Perhaps human consciousness is nothing more than physical interactions of the laws of nature: chemistry, atomic forces, molecular biology, quantum fields, cellular and neuronal connections, and so on. Or maybe human consciousnesses is more than this. Subjective experience, therefore, can't be simulated by a machine, as it is not reducible to physical interactions and there is something more to consciousness that we can't explain sufficiently by scientific explanations alone. (See: qualia and the philosophical zombie argument). It could be due to the fact that our brains evolved to be the way they are now over billions of years, starting from the first single-celled life, and the emulation of such a process is much too complex — although we couldn't completely rule out this possibility.

If conciousness emerged and we can create similar conditions which led to this emergence, by building a machine or computer program that can emulate this emergence, could we then create and artificial consciousness? Very hard to know. If the atoms of an intelligent and conscious thing are silicon-based rather than carbon-based, and are arranged in a precisely identical arrangement, are we talking about the same phenomena? Consciousness in its varied forms?

Searle's comments about this:

"Could a machine think?" The answer is, obviously, yes. 'We are precisely such machines. "Yes, but could an artificial, a man-made machine, think?" Assuming it is possible to produce artificially a machine with a nervous system, neurons, with axons and dendrites, and all the rest of it, sufficiently like ours, again the answer to the question seems to be obviously, yes. If you can exactly duplicate the causes, you could duplicate the effects. And indeed it might be possible to produce consciousness, intentionality, and all the rest of it using some other sorts of chemical principles than those that human beings use. It is, as I said, an empirical question.

If you are of the opinion that it doesn't matter whether the computer 'really' knows it is conscious and can pass a sophisticated Turing test, then appearance is all you need to be convinced that an AI can really be conscious. Strong AI, to Searle, is human consciousness only and can't be replicated by a computer program. Very hard to define what is meant by consciousness, so his entire argument is underpinned by knowing at-bottom intuitively what we mean by "subjective experience of reality". Only you really know what it is like to be you, and no computer will ever be able to really 'know' how it exists in the same way. Without seeing the limits in the future of what our technology can achieve; however, we will not be able to unconditionally rule out a self-conscious super-intelligence or strong-AI.

Reference:

Cole, David, "The Chinese Room Argument", The Stanford Encyclopedia of Philosophy (Winter 2015 Edition), Edward N. Zalta (ed.), URL = https://plato.stanford.edu/archives/win2015/entries/chinese-room/.

Kirk, Robert, "Zombies", The Stanford Encyclopedia of Philosophy (Summer 2015 Edition), Edward N. Zalta (ed.), URL = https://plato.stanford.edu/archives/sum2015/entries/zombies/.

John R. Searle, "Minds, Brains and Programs' inThe Behavioral and Brain Sciences, vol. 3. Copyright @ 1980 Cambridge University Press.

  • So for Searle: understanding > thinking ? If not it seems strange for him to casually say that the question of a thinking machine is "empirical." That's precisely what the Turing test was meant to address. Do you know what he has in mind? – Tim kinsella Jan 20 '18 at 2:43
  • @Timkinsella I think it's a grasp of semantics or understanding the meaning of words, not just 'symbol manipulation' or syntax (rule following) as Searle says. I sometimes think that if a computer appears to understand, then we are perhaps none the wiser. The machine can 'understand' like we do. That is until we get to a point where we can break its programming, maybe feeding it Gödel sentences which it cannot handle. – jphillips Jan 20 '18 at 2:45
  • 2
    I guess I'm wondering, if Searle believes the question of whether a machine is thinking is an empirical one, then he must believe there is a test, superior to the Turing test, which really can sort the thinking machines from the zombie machines. right? – Tim kinsella Jan 20 '18 at 2:54
  • 1
    Ok thanks. I see that the possibility of such a simulation is an empirical question. I would think though that the meaningful part would be discerning whether the entities in the simulation are conscious (or capable of understanding or thinking). – Tim kinsella Jan 20 '18 at 3:04
  • 1
    @jphillips so if we knew the algorithms of human consciousness at a suitably abstracted level (abstracted above the biology of the organic brain) we could realise the algorithms in an electronic digital computer (unless Searle is right and for some unstated (unimaginable?) reason this cannot be done). AI doesn't know the algorithms of consciousness. But what about the structures of consciousness? Surely it need to know these too. There often seems to be a fixation on algorithm (process) which ignores structure. Searle ignores structure. That's what I think is wrong with his Chinese room. – Roddus Jan 27 '18 at 2:52
3

There seem to be several things not understood in asking this question.

Searle gave an intuitive argument. He did not and still does not understand the details so there was a limit to what he could explain.

It doesn't actually matter if you used books or you used a filing system or a database or you used a state of the art AI, the results would be the same. So, whatever Searle said would be right or wrong regardless of how you built it. What would happen if you built such a room?

The room would be capable of answering any stock question correctly. This would be limited only by the size and access time of your information base. BTW, this is pretty much what Watson did when it played Jeopardy. It understood none of the questions or answers but only looked for associations. So, Watson is probably the best example of a modern, Chinese Room. If you asked a stock question like, "What US president was rumored to have chopped down a cherry tree?," Watson would find that US, president, and cherry tree were associated with George Washington and would answer correctly. But this isn't understanding, not even close.

The human mind is strong in terms of its ability to generalize which means applying known patterns and logic to conditions outside of previous experience. You could in fact trap Watson very easily. Let's take

Suppose you were in a room with a locked door. There is a button on the wall that will unlock the door. However, the button is too high to reach even if you jump. There are a sturdy, wooden table, a broom, and a rubber ball in the room. How might you try to escape?

This question is trivial for a human but Watson would be unable to give any kind of answer unless that question and an associated answer was in its information system. I've even heard of experiments that toddlers could solve but Watson would be incapable of. And, this will still be the case no matter how large you make the information base. So, Searle was mostly correct.

  • 1
    I don't now if that's a fair representation of Watson or computers, generally. Certainly no computer works by consulting a lookup table (even in principle) as the Chinese room does. Also, at minimum Watson has to (and does) do lots of natural language processing. I just looked at a clip from the show where Watson correctly answers the question "Gambler Charles Wells is believed to have inspired the song 'the Man who' did this 'at Monte Carlo'". Certainly Watson has the lyrics in his memory, but he still has to "understand" the question, in some sense. – Tim kinsella Jan 20 '18 at 15:13
  • This is a neat video youtube.com/watch?v=DywO4zksfXw . The example they use there is the question "the first person mentioned by name in 'the man in the iron mask' is this hero of the previous book by the same author." – Tim kinsella Jan 20 '18 at 15:34
  • @Timkinsella Watson is a text search engine with a back-end database to increase accuracy. Watson works by finding words and phrases on the internet along with associations and then checking its database. This is exactly like a lookup table. Again, you are greatly overestimating what Watson does. I did a Google search on "charles wells man monte carlo" and the very first hit contains, "The Man Who Broke the Bank at Monte Carlo". You are confusing a keyword search with understanding; Watson has none. – scientious Apr 13 '18 at 17:41
  • @Timkinsella Yes, the video shows how Watson works although when the narrator uses the word 'understand' it does not have the human meaning. The first part is parsing; this is a well known function which is routinely used on compilers. Did you note that Watson tries multiple variations of the question? This is called a brute force search and again implies no understanding. She says, "quantity trumps accuracy". That's the opposite of how a human would approach the question. The description is pretty good but she does use personifying terms like understand and know; these aren't accurate. – scientious Apr 13 '18 at 17:55
  • @Timkinsella In the last part, Watson takes its possible answers and eliminates the ones that are obviously incorrect and then weights the rest based on frequency. It then uses pattern matching to see if the answer style matches the style of past Jeopardy answers. Then it uses standard game theory to calculate the risk and reward of answering which is very routine. In this entire process the only part that required neural network learning was the Jeopardy style pattern. Again, nothing in this process requires comprehension of the material. Watson's lead programmer has said the same thing. – scientious Apr 13 '18 at 18:02
1

Consider the following about the Chinese Room Argument.

First, strong AI is a view that programs running on Turing machines (computers) not only produce correct results but also generate consciousness when run.

Second, assume there exists a program that passes the Turing test for Chinese when run on any Turing machine no matter how advanced or primitive that computer may be.

Third, let a human who does not understand Chinese simulate a Turing machine by following that successful program while being isolated from outside influence in a “room”. If strong AI is correct this should be a way for the human not only to give a correct answer but also to understand Chinese. This program would be a way for someone to learn Chinese.

Fourth, Searle claims the human will not understand or learn Chinese, but the human will be able to pass the Turing test using the program since the program is assumed to be able to do so.

Fifth, look at the program as the “mind”. Look at the computer as the “body”. The body runs the mind and this supposedly generates not only results, but also understanding of Chinese. It is this mind-body dualism that is the problem for a physicalist.

Given the above, I will try to answer this question.

“Humans first have to learn Chinese. Why doesn't the room try to learn Chinese?”

The room is only a way to isolate the human from outside influence. It is not the computer. There is nothing for the room to learn. The human needs to learn or understand Chinese by running the program. Running the program is what counts not which computer is used to run the program. The program can be moved around to different computers or different humans and the results should be the same. The question is does running the program anywhere generate not only a correct result, but also understanding of Chinese? If it does, then this would be an alternate way for someone to learn a new language.

Reference: John Searle, "Minds, Brains and Programs"

  • So what sort of program would need to be run in order to learn a new language? Would this program contain examples of the symbols of the new language? If so, how could running the program be learning a new language (i.e., one the system has never come across before)? – Roddus Jan 22 '18 at 4:07
  • 1
    @Roddus We learn a new language by associating the meaning of the words with the sounds or symbols. Learning will need to incorporate meaning and not just symbol manipulation. The programmer who knows the language tries to find a way to communicate without the meaning, but only the syntax of the language. Meaning is a subjective experience of language, but syntax alone is objective and does not require a subjective experience of the language. To go from the objective to the subjective is why strong AI will not be successful and Chinese will not be learnt by running the program. – Frank Hubeny Jan 23 '18 at 3:12
  • We associate meanings with symbols/sounds - OK. For learning will need to incorporate meaning not just [perform] symbol manipulation. Yep.The programmer tries to communicate without the meaning but only syntax. OK - so he/she uses syntax to create machine behaviour that humans can produce using meanings? So syntax is a sort of substitute for meanings, or is derivative of meaning but itself is devoid of meaning? Syntax can be used to produce the right bahaviour but that's all? – Roddus Jan 25 '18 at 2:49
  • @Roddus That is sort of how I see it. I think of meaning as subjective, a bit different for each person. Abstracting that away makes the syntax objective--good for all persons. Still each person understands the syntax differently as well. That's why it is good for machines to manipulate the syntax, again, to remove the subjective differences in understanding syntax. Understanding is subjective and different for each of us. If we remove it to get the machine to perform objectively we can't bring it back without having a person with subjectivity getting involved. – Frank Hubeny Jan 25 '18 at 18:25
1

I believe that you've hit on what is typically referred to as the "systems reply", which is, in short, that the room system does understand Chinese. This seems plausibly true in terms of a functional definition of understanding, but getting to the point where one can conceive that the room system has a subjective experience of understanding is a much bigger conceptual chasm to jump. (though as far as I can tell, it cannot be ruled out, especially since I don't know to a high degree of certainty which of you dear readers have subjective experiences and which of you are zombies).

  • About subjective experience, that seems right - a big jump - the idea of subjectivity seems actually quite ill-defined, too, though the distinction between objective and subjective in some contexts is important. On the systems reply, Searle says, well, there is nothing in the room that could conceivably amount to a semantics. I agree with this. But I argue that there should be more in the room, namely relational elements (eg nodes, connections). Computers have these. They could be used to build structures as might embody learning. Neural net structures learn patterns from data sets. – Roddus Jan 27 '18 at 0:25
  • 1
    @Roddus The problem with paying attention to the room or the computer or the human is that it no longer pays attention to the program which is what is supposed to generate understanding. In the CRA we have the perfect "computer" with everything we need for understanding because a human being is instantiating the program by running through the program's steps. In spite of this, the human is not expected to understand Chinese through this method of learning the language. That neural networks "learn" is metaphor. – Frank Hubeny Jan 27 '18 at 1:02
  • @Frank Hubeny I agree as per searle's description of the room that the mind is (according to strong AI) embodied in the program - the mind is to the brain what the program is to the computer. BUT the human brain is full of structure. Processes propagate from place to place through the structure. The program is just the process. How would the structure of the human brain be embodied in the computer program? If it can't be then presumably the Chinese room needs something to relate symbols together in to structure. Or do you think this is not needed? – Roddus Jan 27 '18 at 3:13
  • 1
    @Roddus That's perhaps one reason why Searle objects to strong AI--it is not adequate to explain the mind. He also doesn't like the mind-body dualism implied in strong AI. If the mind is a program one can move it from body to body and duplicate it. However, one could object to his physicalism in a similar way he objects to strong AI by claiming it also is not a complete explanation for the mind. So even giving up on strong AI and adding stuff to the room to better simulate the brain is not enough to generate understanding or consciousness. – Frank Hubeny Jan 27 '18 at 13:24
  • 1
    @Roddus As I understand "program" it is anything on that theoretical tape a Turing machine reads which would include arbitrarily large memory. However, I agree that a real program seems separate from the physical memory it references. Don't forget, when the human traces through the program he has access to his own memory so the added program should be all he needs if strong AI is true. Your objection to the brain structure not being part of the mind is how I read Searle's objection to strong AI. He thinks strong AI is missing too much by focusing only on computation. – Frank Hubeny Jan 28 '18 at 15:00
0

This question comes down to qualia, is there a difference between a state as observed from outside, and subjectively from inside. The Chinese Room helps us thinking about that, but raises more questions than it answers.

We are moving into a new era with this question however, as we approach the point of being able to build such a room.

From the ground up, we have natural language processing like https://en.m.wikipedia.org/wiki/Watson_(computer)

And from simulating the human brain down, we have https://en.m.wikipedia.org/wiki/Human_Brain_Project

A human, or a room that has 'learnt Chinese' rather than just been given limited responses to preprogrammed situations, would not only give answers to well formulated questions, but be able to interpret incomplete or unclear questions, and reinterpret definitions on the go (Watson struggled with short questions, for instance, lacking context). The setup tries to say these wouldn't, but natural human language is full of them. Yes the room needs structures, parallel to human mental structures or learning, and will only be as capable as they are sophisticated. We have to look at the threshold between rote learning and understanding a language, like between brute forcing chess moves and being able to understand the game. What is understanding?

We do complex things, and take them for granted, which muddies the water of this, similar to how we found computer visual processing to be a great deal more difficult than expected, because so much is happening we aren't aware of. The best framework I know for understanding what may be making the difference, from the ground up, is the https://en.m.wikipedia.org/wiki/Strange_loop picture.

  • It's a great point about incomplete questions that a semantic system would be able to adequately answer. So Searle's idea that the program in the room is so good it can adequately answer any question has a problem. How could it answer incomplete questions without understanding the context? Could there be such a thing as a program that could adequately answer incomplete questions? – Roddus Jan 29 '18 at 23:02
  • Your point about brute forcing chess moves compared to understanding the game: the program in the Chinese room brute forces an answer to the Chinese question (and has no understanding of the language) but a human understands the language. What is understanding? Well I suppose one idea is connections between internal representations, i.e., that the meaning of a symbol (word...) is a group of associated internal representations. – Roddus Jan 29 '18 at 23:12
  • 1
    And about your point of rote/brute force verses understanding. What about this question for the Chinese room (in English): Take these two questions: (1) the police arrested the protesters because they were drunk, (2) the police beat the protesters because they were drunk. In these two questions, who does "they" refer to? What answer is the rule book in the room going to give? And what about the question, Which of these names would be best for a kitten: grax, hrip, mooggle, dast, lagz? – Roddus Jan 30 '18 at 0:08

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.