1

For those who don't know, here is a description of the Chinese Room Argument.

The argument is essentially that even if an AI may give the impression of being intelligent because they answer questions correctly they could still just be a better dictionary that maps inputs to outputs without understanding why the inputs and outputs are connected. Like if you'd trap a monolingual non-Chinese speaker in a room, give him instruction in Chinese and a rule book that connects input shapes to output shapes. So despite the system of the "Chinese Speaking Room" giving correct answers to the Chinese questions, thereby convincing Chinese speaking bystanders that it has understood the people inputting questions, the person inside still has no clue and no way to figure out what any of the symbols, both the input and the output, actually mean. And therefore no clue what it is doing. Like it could be translating poetry or hate speech and it wouldn't know the difference.

But what it misses is that the person inside the box is still a human/strong AI, it's just artificially limited by the input/output system. It can only follow the rules and be perceived as working or break the rules and be perceived as defunct. So it's less of an argument against the existence of a strong AI and more of our inability to realize it when we see it.

I am well aware that the person inside the room is just a human/strong AI to make the point that "even if we gave that thing all that we could think of having it would still not be able to learn Chinese like that". But nonetheless it is a strong AI. So technically we could still argue that we could build a smart AI trapped in a dumb job, that to bystanders happens to look smart.

And this actually creates a problem for the replies to that argument. Like for example the "Robot Reply". That asks: "What if the room isn't a black box but a robot that can see and move". Then from making a move and viewing the result the robot or rather the person inside the robot would be able to gain awareness of their movement and from that maybe spacial awareness and even "self-awareness" aso. And that is not a novel experiment, we regularly do that in computer games.

Like you could switch the perspective and argue that the computer game is the real world and that we are the simulation and by press a button and seeing the result of that on the screen we are able to gain awareness of "ourselves" in this world. Or for those less inclined to video games, maybe technical jargon. It's a different language there are surprisingly many people who have no idea what the terms actually mean because they often happen to be in a different language but by being aware how they are used they develop a practical understanding of the terms that is often indistinguishable from actual "knowledge" of the language.

So in reality we still don't know Chinese (how the actual movements of "our body" work), but we know how to cause them and how the environment reacts to it and that is already what we call "understanding" isn't it? Or what is the difference of knowing the meaning of a word and knowing it's effect?

So again to stress that point, I know that I'm being a little facetious about this thought experiment and that this wasn't really what it tried to prove, nonetheless am I missing something? Or isn't it just artificially incapacitating the AI in order to prove it's not smart?

Improve this question
10
  • what is the question here?
    – Stacker Lee
    Jul 20, 2022 at 23:12
  • 2
    You are indeed missing the point, as did Searle. It is exactly because we don't know what "intelligence" is that Turing resorted to the black box experiment: We know it when we see it, so let's observe it and see. If it looks intelligent, smells intelligent, acts intelligent: Then for all intents and purposes it is intelligent. As a more concrete reply one could argue that our brains are essentially "Chinese Rooms", consisting of a team of cooperating "agents" none of which on its own is "intelligent", yet we produce answers that convince others that we are.
    – Peter - Reinstate Monica
    Jul 21, 2022 at 9:02
  • Or, maybe (and this is your point): One of the "agents" (subsystems) in our brain could qualify as "intelligent" on its own: That is entirely irrelevant.
    – Peter - Reinstate Monica
    Jul 21, 2022 at 9:03
  • @StackerLee Whether that question actually proves that there cannot be a strong AI when it literally premises a strong AI hiding in that black box and we couldn't tell.
    – haxor789
    Jul 21, 2022 at 10:11
  • @Peter-ReinstateMonica So "intelligent is what does intelligent things" and "what things are intelligent are determined by things that are intelligent (us)"? Yes I see how that is a flawed definition. But that doesn't explain why the agent in the box can't be intelligent or why that would be irrelevant. I mean as far as I understand it that's what he wanted to prove right?
    – haxor789
    Jul 21, 2022 at 11:14

2 Answers 2

Reset to default
5

No, there’s no artificial incapacitating. The rules are set up in a certain way to draw certain conclusions. The Chinese room works for all computation at any level. The outputs can be as sophisticated as the best native Chinese speakers. It’s less a measure of ability, rather computational ability does not elucidate the mind very much.

Digital computer programs are specified with purely formal syntax and that syntax has zero semantics according to Searle. The goings on in the room are purely computational (save part of the mind of the non-Chinese monoglot inside if that version). Therefore at the end of the day, no matter the program’s technological power, it is purely formal, a series of statements with only syntactical structure.

Well Chinese speakers do NOT understand Chinese purely formally. Therefore the monoglot’s mind does not understand Chinese like a Chinese speaker does. And certainly neither does any other part of the room nor the whole room (no strong AI). Every input and instruction of the room is purely formal. The computer in executing formal syntax has nothing you don’t have. Therefore if you can’t understand Chinese like a Chinese speaker while in this room, no computer can either.

You might say well even if syntax alone can’t provide human understanding, maybe it provides formal understanding. But would you say the same thing for an abacus or a handheld calculator? Searle’s room works regardless of the technological and computational level. Formal in->formal out regardless. It diagnoses all computation at once.

Mental states demonstrate and require more than formal properties.

Improve this answer
8
  • 1
    You seem to use a very different definition of "syntax" and "semantics" from what computer scientists use. The computer scientists define "syntax" as the set of rules for formulating valid code in the language. It is the parsers' job to check whether a given code actually conforms to the syntax. The result is a simple "yes" (producing an abstract syntax tree) or "no" (producing an error message). A compiler will then analyze the abstract syntax tree for its formally defined semantics (the behavior of the program as described by the code) to translate the code into machine language.
    – cmaster - reinstate monica
    Jul 21, 2022 at 1:07
  • 1
    @J Kusin: "Mental states demonstrate and require more than formal properties." So what would prove or disprove (at least to some extent) that a computer or other artificial/inorganic machine, was or was not having "mental states"?
    – The_Sympathizer
    Jul 21, 2022 at 2:42
  • Did he produce a proof as to why formal languages alone cannot produce semantics? Or is that example meant to be the proof?
    – haxor789
    Jul 21, 2022 at 6:47
  • @cmaster, he's using the words more generally. In CS, syntax and semantics have precise meanings because of programming languages, but more generally syntax refers to text and semantics refers to meaning. Also, meaning is not a formal entity like a transformation from a language to a virtual machine; meaning is something that is apprehended by a mind when the mind understands a communication. Understanding a meaning is a subjective experience. This distinction is often lost by people who dive down the layers of abstraction too quickly.
    – David Gudeman
    Jul 21, 2022 at 8:43
  • @DavidGudeman "Digital computer programs are specified with purely formal syntax and that syntax has zero semantics" - Kusin is using the terms "syntax" and "semantics" in the context of computer programming here. As such, it's a reasonable expectation to assume the precise CS definitions of these two terms in this context. An expectation that is sadly not satisfied.
    – cmaster - reinstate monica
    Jul 21, 2022 at 9:00
3

I think, the key take-away from this thought experiment is simply that we still have no clue what it means to "understand", be "intelligent", or "conscious". So much so, that we are unable to recognize it, or even to avoid being fooled into thinking something is "intelligent" when it's actually not. We are even unable to precisely define what "intelligent" means.

What we can do, is to

  • Analyze the workings of the human brain in a bottom-up way, in the hope of understanding how it generates what we call "self-conscious intelligence". We are only at the very beginning of this endeavor.

  • Build ever more complex computer systems and observe in which ways we can call their behavior "intelligent" or "self-conscious". However, all the systems we have built today, can be proven not to be "self-conscious" in any way. And most fail the "intelligent" trait in interesting ways.

In both approaches, precise definitions of the terms given above would be a result of the research, not an input. And we are still very far from getting these results while we are still trying to understand all of the chemistry in our brains. The traits of intelligence are an emergent phenomenon from the chemistry and precise structure of our brains (connectome) that we may never fully understand. Whatever it means to "fully understand" something.

Improve this answer
14
  • People have a very clear understanding of what it means to understand, be intelligent, or be conscious. They prove it by using the words in successful communication every day. What they don't have is a reduction of these things to something else, but you don't have to be able to reduce one thing to another in order to understand it.
    – David Gudeman
    Jul 21, 2022 at 8:47
  • @DavidGudeman Do they? Can you prove to me that you understand what I'm saying, and not just algorithmically construct an answer using all the texts on the internet and, especially, the wikipedia? Can you prove to me that you are conscious? All I know is that I'm conscious, because I can experience myself thinking. Like Descartes said, I think, therefore I am. But I am unable to prove it to you. We can both only assume that the other has a similar experience, and therefore assume that the other is conscious. But neither can be certain of it in a fundamental way.
    – cmaster - reinstate monica
    Jul 21, 2022 at 9:08
  • I don't understand your point. I said people prove that they understand certain words by using the words in successful conversation, and you seem to be challenging this notion on the grounds that you don't think I can prove that I'm conscious. Of course I can prove that I'm conscious, but I don't see the connection.
    – David Gudeman
    Jul 21, 2022 at 10:34
  • As to whether I can prove that I'm conscious, of course I can, at least in the normal sense of proving things--I can prove it beyond any reasonable doubt. Can I prove it beyond any possible doubt? Of course not, but practically nothing can be proven beyond any possible doubt. I can't prove beyond any possible doubt that the screen you are typing on is still there when you can't see it. I can't prove beyond any possible doubt that you aren't living in an artificial world like the Truman Show where everyone you know is out to fool you. It's not a tenable standard of proof.
    – David Gudeman
    Jul 21, 2022 at 10:41
  • @DavidGudeman People can develop some sort of understanding what these words mean, but more or less by negotiating a practical definition for the immediate usage. But beyond that each of these words has different facets and degrees that if you'd try to overlap them wouldn't really produce a useful and clear definition of the thing would they? So what is it that people understand here? Like if you chain some homonyms you might think you understand but in reality you're talking about vastly different things.
    – haxor789
    Jul 21, 2022 at 11:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .