2

The following argument concludes that the common understanding of ChatGPT (trained on text, receives online users' text questions, etc.) is not supported by the science. What criticisms are there of the argument which show it is unsound (one or more false premises) or invalid (breaks the rules of formal logical deduction)? The argument is:

In November 2022 AI research firm OpenAI released its online "text" chatbot ChatGPT. The following great interest was based on the popular and expert-led conviction that ChatGPT did, or with suitable (multi-billion-dollar) development might, understand the text questions online users type.

ChatGPT exists inside the electronics of computers. The research I conducted indicates that the science of the physics and chemistry of electronics doesn't support the idea that ChatGPT could understand online users' text questions. The reason is as follows. The only place text exists is on user keyboards and display screens. Everything else is electromagnetic radiation (WiFi, Bluetooth, fiber optics) and electrons (in wires and semiconductors).

ChatGPT never gets the text questions online users type. All it gets is electrons. The internet is made of electronic storage. All that's stored on the internet is electrons. There's no text on the internet. The ubiquitous claim that ChatGPT is trained on text stored on the internet is simply false.

Relevant quoted and referenced statements of OpenAI and respected media allegedly falsely describing ChatGPT can be found in Does ChatGPT understand text (PDF).

7
  • Comments have been moved to chat; please do not continue the discussion here. Before posting a comment below this one, please review the purposes of comments. Comments that do not request clarification or suggest improvements usually belong as an answer, on Philosophy Meta, or in Philosophy Chat. Comments continuing discussion may be removed.
    – Philip Klöcking
    Feb 25 at 19:26
  • 12
    A completely bizarre analysis. You might as well argue that there is no text in a printed book, only splodges of black ink splattered over the paper. Feb 25 at 23:15
  • 3
    "All that's stored on the internet is electrons." All that comes into your eyes is electrical impulses generated by your retina / optic nerve. I think you are getting hung up on the word "text" as a physical thing rather than information, in whatever physical forms it may take or be transformed into.
    – user4574
    Feb 26 at 3:26
  • Nobody who understands what LLMs are has ever genuinely thought they understand anything... They are designed to predict what the next word would be based off of the input and training set. That's clearly not the same as understanding the underlying concepts. Just ask it to do some basic maths and you'll see this. That being said, this analysis is ridiculous... Feb 26 at 10:51
  • 1
    @CriglCragl Thanks for the support. I've edited the question and removed references to myself and also made the question more formal. Hopefully that will meet the tests. I was looking for criticisms of the argument. I wanted to be told them and to understand them. So hopefully the question will be reopened.
    – Roddus
    Feb 29 at 6:10

9 Answers 9

15

The argument is irrelevant. You can apply the same logic to say that humans don't understand text, since text doesn't exist in your brain. There are lots of software applications that read text- if you integrated one with ChatGPT, would that mean the integrated system suddenly 'understood' text in a way ChatGPT did not? You can tell Rodsmith from me that the argument is nonsense, and he should present himself at my study if he requires further clarification.

11
  • You say, "There are lots of software applications that read text". I find this is one of the strangest things much like the denialism following disconfirming instances of phlogiston theory. My background is in software development and I found it really difficult to overcome the view of computers as symbol (i.e. text) processing systems. Understanding Turing machines helped. They internally manipulate text (0, 1, x, y, ::, etc.). Everyone says computers are Turing machines. They're not. Computers internally manipulate electrons. Electrons are not text. They're sub-atomic particles.
    – Roddus
    Feb 24 at 21:39
  • 6
    You cannot use vehicles to go from one place to another because vehicles are nothing else than protons and electrons. Would you agree? Probably not. Similary with computers. They don't internally manipulate electrons. We build our computers, for some historic reasons, as electronic circuits that use various effects of electricity but all our computers manipulate are their own internal state. The computer is only concerned with this state and not the underlying physical processes we use to build them upon.
    – Gábor
    Feb 24 at 21:48
  • 9
    @Roddus you are fixated on the visual appearance of text, as if it were important. It's not. Blind people can learn to read without seeing text. The symbols are just that- symbols. When people say that CGPT is trained on text, they don't mean it is trained on the visual appearance of printed text. Feb 24 at 22:20
  • 1
    Suppose then that I compose a delightful sonnet in my head- where is the text of the sonnet? Feb 25 at 7:11
  • 3
    "You can tell Rodsmith from me…" I suspect you already know this, but for other readers it might be worth noting that the OP actually is Rodsmith (as can be seen in their bio page). The question is a strange attempt at self-promotion. Feb 26 at 5:59
12

ChatGPT never gets the text questions online users type. All it gets is electrons. […] The ubiquitous claim that ChatGPT is trained on text stored on the internet is simply bonkers.

This argument is based entirely on a misinterpretation of what the word "text" means in this context.

In computing, "text" means a sequence of codes which are designed to represent written words, essentially by using one code for each character in the writing which is to be represented. It is obvious that ChatGPT is trained not on text in the sense of matter physically arranged into written words, but on text in the computing sense.

You're a software developer, so presumably you're familiar with the use of the word "text" to mean a sequence of character codes. It's not clear why you chose to interpret the word "text" in a way that produces a "bonkers" claim instead of a sensible one.

The argument says nothing about ChatGPT's ability or inability to understand these sequences of codes, so there is no need for me to say anything about that either.


By the way, I notice that the idea that symbols are shapes rather than other types of patterns seems to be very important to you. In this question from 2019, you asked why Alan Turing described computers as manipulating "symbols," given that the things that they manipulate are actually voltages and not shapes. In this question and this question, both from 2022, you point out that when people do what they call a Turing test in real life, the machine almost always receives electronic pulses instead of shapes, and you assert that "[w]ithout exposure to the shapes, the machine could not possibly understand the questions."

Why do you think that the distinction between shapes and other kinds of codes is important? When people listen to each other speaking, they don't hear any shapes, they only hear information about frequencies and amplitudes; does that mean that it is impossible for any human to ever understand speech? If a person has never seen any shapes representing words in their life, does that mean that that person is unable to understand language at all?

It seems like you keep repeating the argument "computers don't receive shapes as input, therefore computers can't understand anything" over and over, presumably hoping that someone will eventually agree with your argument (or at least understand it), but you've never given any reasoning for why you think the argument is valid. The argument seems like a non sequitur to me.

4
  • Electronic engineers know that no text (ABC123...) is received by or processed inside computers. I argue that ChatGPT is said to receive such linguistic text. Why? Because, it is trained on text ("books", "articles",...) found on the internet. That's what's commonly said. But you argue that the term "text" when applied to computers means groups of electrons (clocked or in semiconductor traps, etc.). I've worked with assembler and C and not heard of his meaning. But assuming it exists, have Sam Altman and others explained that by "text" they mean something totally different from everyone else?
    – Roddus
    Feb 24 at 23:03
  • (Cont...) "I want to distinguish between shapes and voltages for a semantic reason. We understand the meanings of shapes (as in reading a textbook). We don't understand voltages (have no clocked-voltage sensing modality).
    – Roddus
    Feb 24 at 23:35
  • (Cont...) You say "It seems like you keep repeating the argument "computers don't receive shapes as input, therefore computers can't understand anything" over and over... [it seems] like a non sequitur". My argument is a bit different. Searle's CRA says computers just receive interpretable shapes, and such shapes in themselves mean nothing (interpretation being extrinsic), and all a computer gets is the shapes, hence no computer could understand anything. I reject this argument concluding computers, once realizing principles of knowledge acquisition via sensory apparatus, could understand.
    – Roddus
    Feb 24 at 23:46
  • 5
    @Roddus "I've worked with assembler and C and not heard of his meaning." – You've never heard anyone describe a file as containing "plain text" before? The official documentation for C, Python, and HTML all describe those languages as consisting of or handling "text," meaning sequences of character codes, not text in a physical or visual form. If you've never heard the word "text" used in the way that those sources use it, I can only say that your experience has evidently been very different from mine. I apologize for not responding to your other comments immediately; I'm short on time. Feb 25 at 2:27
7

First of all - LLM has no conscience so one can argue they do not understand anything at all. Does a flashlight understand that you switch it on when you push a button on it? No, it's just turned on cause there is electricity that starts flowing. If we continue this line of reasoning why does it matter at all in which form LLM process data? We shouldn't anthropomorphize LLMs as consciously understanding. They have no idea they even exist not to mention text understanding.

On a more technical note - the paper you provide leans quite heavily on Searle's CRA, but does not address counterarguments that have been made against it. The thing is reference, and semantics are not intrinsic qualities of arbitrary symbols. They emerge from the relationship between internal states of an agent and its external environment, through the intercession of the senses. Symbols must be "grounded" in sensorimotor interaction with the world to take on meaning.

The Chinese room, for example, cannot claim genuine semantics because it lacks grounding - its symbols derive from rules given by external designers, not internal perception and action. The manipulated symbols are not meaningful.

By relying on intrinsic semantics, the CRA ignores the actual relationship between representation and reference that gives rise to true semantics. That's a weakness.

9
  • 2
    +1 Out of curiousity, what constitutes "genuine semantics"?
    – J D
    Feb 24 at 15:54
  • I agree with the need of grounding via sensorimotor apparatus, and Harnad's arguments are well known. But I see the CRA as having two conclusions. The first is sound (has true premises) but the second isn't. The first argues no computation, the process, could understand what it receives and internally manipulates because these items ("symbols" meaning text) have no intrinsic semantics. They do not carry, contain or indicate their meanings. But the second argument has two false premises: (1) that computers, the hardware, manipulates text, (2) and do nothing else. The CRA does ignore grounding.
    – Roddus
    Feb 24 at 21:22
  • (Cont...) but I don't understand "the actual relationship between representation and reference that gives rise to true semantics". If by "representation" one means an internal structure, and by "reference" one means things in the outside world, then some relationship between the first and the second seems to be the classical idea of how words mean things (I might have this wrong). However a full understanding of semantics will presumably explain why solipsism can't be rebutted. There might be no external world at all. Semantics will need to encompass more than just the inner-outer relation.
    – Roddus
    Feb 24 at 21:31
  • @J D, "Out of curiousity, what constitutes "genuine semantics"?". In short, I think a place to start considering semantics is with understanding (a process), knowledge (a structure) and intentionality (a property of the structure). Of course this needs explanation. Geoff Hinton recently said "semantics, whatever that is". Greg Brockman said as much. They don't know. Yet thinking, the semantic process yielding understanding, is key to intelligence. In other words, the field of AI research doesn't really know what intelligence is. But structure, process and property seems a place to start.
    – Roddus
    Feb 25 at 1:34
  • Or maybe understanding has nothing to do with thinking, maybe it’s not verbal but visual? Do you see what I mean? Feb 26 at 8:43
7

People are quick to point out that AI is merely electricity passing through some wires and transistors and such and to perform basic mathematical operations, and therefore AI is not intelligent like humans are.

But these arguments generally seem to take the form of "AI is like this, therefore it's not like humans in terms of understanding", without actually addressing what humans are like. That's comparing 2 things by only analysing 1 of those things. Considering the mechanics of human intelligence (as best we understand it) doesn't seem to end up in favour of those saying AI doesn't "understand" or aren't "conscious". Human brains are, after all, a bunch of connected neurons passing electrical signals between one another, in some chemical soup. Some might say there's more to human consciousness than our brains, but we have no evidence of this, and every indication that consciousness is strongly and directly tied to the physical parts of our brain.

It's probably fair to say the physical parts of the brain is more than just computation (which arguably set it apart from modern AI), although we're not currently able to say which mechanisms in the brain makes a human conscious, and how it does so. Even the people who says the mind exists beyond the brain still can't describe the mechanism for understanding or consciousness within such a mind (nor describe how such a mind would interact with the brain on a mechanical or logical level).

Knowing a thing or two about AI doesn't make someone an expert in neuroscience, yet some people act as if it does, as if they have a perfect understanding of the mechanisms of "consciousness" and "understanding" in humans, for them to be justified in saying AI isn't conscious.


This "it's all electrons" seems to be taking a few steps up on the scale of smart-sounding nonsense.

One could say "ChatGPT never gets the text questions, because it just gets electrons", but what does it even mean to "get" something? You could similarly say that your brain doesn't "get" the things you see or hear because all it receives is electric signals from your ears and eyes via your nerves, and the brain itself works largely by electric signals being passed between neurons.

* I briefly scanned through the paper. It does mention the need to "chose[sic] a place to start trying to understand the mind", but it's hard to make sense of the gibberish that follows as anything that could be said to differentiate human minds from AI.

8
  • saying "it's all electrons" seems to be taking a few steps up on the scale of smart-sounding nonsense". Well at least it's smart-sounding. But the point is that ChatGPT gets electrons, that's all. Virtually everyone says that ChatGPT does or might understand what it gets. I press my keyboard's character shapes. I click Send. Electrons or EMR go out. A human email recipient sees the same shapes. They react to them and understand what was in my mind. ChatGPT receives electrons. It doesn't react to shape. How does it know what was in my mind when I pressed my keyboard keys?
    – Roddus
    Feb 25 at 1:59
  • @Roddus From some brief research, when you "see" something, the input your brain receives via nerves is ions. The input ChatGPT receives is electrons. Those are similar enough that I don't think you can say one understands and the other doesn't based on that. That's why the "it's just electrons" argument fundamentally doesn't work.
    – NotThatGuy
    Feb 25 at 19:48
  • 2
    @Roddus "How does [ChatGPT] know what was in my mind" - how do you know a person knows what's in your mind? They respond appropriately and you can ask them questions to investigate. You can do the same with ChatGPT. From that, you might conclude its dynamic long-term memory is functionally non-existent, and it often doesn't respond appropriately. But that applies to some extent to people too, and the next generation of AI might not have those problems, at which point it would be much harder to argue that AI doesn't "know" things.
    – NotThatGuy
    Feb 25 at 19:48
  • @Roddus You don't think in letter shapes you think in the images that are encoded in these letter shapes. The encoding is different you see letter shapes while ChatGPT encodes .... well technically strings of chars, which stripped down are just patterns of 1s and 0s so think of it as something like Morse Code. Actually Morse code is a pretty good example because you don't think of it as beeps and pauses, it could also be light flashes and darkness it could be a person blinking or keeping their eyes open... There are million ways to impliment the concept.
    – haxor789
    Feb 29 at 10:10
  • @haxor789 This is the basic problem as I see it: When 2 humans communicate by email, the sender has a certain understanding in their mind. They press keyboard keys inscribed with certain shapes. The shapes don't contain or indicate the understanding (per Searle's CRA etc). Electronic transmission occurs. The receiver's device displays the same shapes on its screen. The receiver sees the displayed shapes and understands what they mean. There is a meeting of minds. The receiver understands the message. ChatGPT never sees the sender's shapes. How does it understand the message?
    – Roddus
    Mar 2 at 1:56
5

First of all, I'm involved in NLP, so thanks for the paper. Some questions that come through here are malformed grammatically, so it's always a pleasure to find a semantically apropos, well cited question.

Obviously, it all hinges on what it means "to understand". The questions about the transformer model and their implications about semantics is a serious philosophical matter if you are willing to move past the claim computers only twiddle bits. The gulf between AGI and AI is best outlined in the recent book The Myth of Artificial Intelligence by Erik J. Larson (GB). Essentially, LLMs and other ML strategies have human-like abilities, but they don't rise to the level of general intelligence.

The point Larson makes is that big data methods like LLMs (which are statistical language models that rely on massive corpora) are inductive techniques and that whatever general intelligence is, it at a minimum is something that integrates deductive, inductive, and abductive methods. As he points out, there isn't even a theory to put to empirical test. In theory then, it's open philosophical as well as scientific question if general intelligence can be exhibited by non-biological systems of computation. (See the SEP's Physical Computation for a thorough treatment of what computation is best described as.)

Thus, it's strong to claim that computers do not understand with general intelligence. Note, there is no firm argument that computers cannot understand. John Searle's argument is considered by many thinkers to be severely deficient. For a good introduction to the criticisms thereof, see Preston & Bishop's Views into the Chinese Room. If you believe that the systems response carries weight for instance, then there is no serious argument that general intelligence cannot emerge in a significantly complex, non-biological system as far as I have found.

Does ChatGPT only get and process electrons? No, this claim is based on a category mistake. Electrons are ontological primitives in the discourse of electrical theory. ChatGPT is a statistical engine that processes vectorized, arbitrary token sequences based on frequency. Anyone conflating those two domains of discourse doesn't understand the basics of how reductionism in scientific theory (SEP) works. In fact, human understanding is built on processing morphemes (mostly) deterministically and contextually in the bigger picture of discourse (See dynamic semantics (SEP) for some insights there), and LLMs do not have access to that sort facility, even if you plunk an LLM into langchain and add some lightweight prompt engineering, use embedding strategies, etc.

Be cautious of anyone who philosophically abuses metaphysical reductionism and who couldn't explain in plain English what a transformer is. If you really are intrigued by the topic, I would encourage you to also read natural language ontology (SEP), categories (SEP), and concepts (SEP). At a minimum, the mimicry of language with formal semantics and human-level intelligence and natural language use are easy to confuse, but also easy with the right technical background to expose as distinct categories.

See also:

1
2

I see from OP's profile that they wrote the paper!
It defines what they understand by "text".

4.1 What is text?

Text is written or printed language such as we find in textbooks. It comprises instances of shapes. But not every instance of a shape is text. A shape is text if it has been assigned a meaning, or interpretation. This is often done by a community.

Such shapes might be atomic such as A, B, C, D and 1, 2, 3, 4, that is, letters of an alphabet and digits. Or they might be such shapes sequenced into words, multi-digit numerals, and so on.

Types of text characters include the just-mentioned words and numerals, and also punctuation marks, special characters such as %, &, @, mathematical symbols, +, -, ^, = and others. In short, the shapes one finds imprinted on the top surfaces of keys of a computer or typewriter keyboard.

To understand text is to know the meanings of its shapes. Understanding text presupposes learning the meanings of the shapes. This is what many enthusiasts think ChatGPT does. They say it's rained on text scraped from the internet and that this training results in ChatGPT learning the meanings of the text shapes.

The last sentence undermines the whole premise of the paper:

The text shapes are not what is transmitted to ChatGPT. It is the electronic standard representations of the text, and the shapes of the letters are defined in the webpage style section (or overridden by the reader). ChatGPT won't care what font the webpage's author preferred the text to be read by humans. It is irrelevant. It is still text.

It is a symbolic representation of text, as defined more than 60 years ago with ASCII, and updated more recently with variants of Unicode. But it is still a symbology of the text, and not the actual shapes.

So ChatGPT is not sent shapes, but a representation of the text which allows it to process the semantics. Human readers are not sent shapes either, but instructions as to what shapes their browser should use, to make the text readable by humans. But ChatGPT does not convert the text representation to shapes, and then apply an OCR (optical character reader) to convert the shapes back to a simpler representation of text which it can process, which it was sent in the first place.

The author might just as well have written a paper explaining that Morse code wasn't about text, because it wasn't transmitting actual "letters" and "words" but pulses.

But they overlook the semantics of what the pulses, electrons, whatever carry, which is contextual (pun intended).

1
  • And there's a second error that's probably even bigger: "this training results in ChatGPT learning the meanings of the text shapes." Nobody ever claimed that ChatGPT learned the meaning of anything. Some tabloid papers might have written something like that but nobody involved with the system ever claimed anything even remotely similar. ChatGPT has no connection to the meaning at all.
    – Gábor
    Feb 26 at 9:36
2
  1. The question whether ChatGPT understands text has the pragmatic answer "Yes": Every user can confirm this by entering text into the text-interface of ChatGPT.

  2. The deeper question is: Does ChatGPT understand the meaning of text?

    Here is the answer from ChatGPT itself:

    As an AI language model, I don't "understand" text in the same way humans do. Instead, I process and generate text based on patterns and associations learned from vast amounts of data during training. When you ask a question or input text, I analyze it, identify patterns, and generate responses based on those patterns. While I can provide coherent and contextually relevant responses, I don't have consciousness or subjective understanding like humans do. My responses are generated algorithmically based on statistical patterns in the data I've been trained on.

    The answer shows what ChatGPT is able to achieve and where it (i.e. it's designer team) locates its limit.

1

The old "It is nothing but a stochastic parrot" argument. Now it is true the main building blocks of ChatGPT and other LLMs do just that: User their neural net to predict on what word comes next. Not much intelligence there.

But then again: This is also a big building block on how our brain works: We lookup things in our own neural net: You are presented some new problem and immediately (without "thinking") some ideas come to your mind. People call it "intuition" or "experience". You hear a word and you find one that rhymes with it and you call it "creativity". Is there conscience? No. We do not know how we arrived at our "intuition" and neither does the LLM.

Now it is easy to see how these building blocks can be arranged to "mimic" conscience: Feed it some input and let it reflect on it. Let it checks the consistency of its results. Let it create new associations from that. Feed it back and let it reflect on it again, etc, etc..

Is this conscience? Well it would certainly look a bit like it. You could ask such a system what it was "thinking" and it could truthfully tell you that it was currently reflecting on some logical errors it found in the writings of Thomas Aquinas and then used its time to search the internet to find places where someone else spotted that but was not successful in finding much.

I think even a system that is based purely on text can get pretty far. Can you understand our 3D reality? At least such a system can reason about all its properties as it knows mathematics. Just as the human mathematician can reason about abstract 5D spaces. But can it understand what it means to enjoy a sunset on the beach when it has never "seen" a sunset? Leaving aside the fact that newer LLMs are "multi modal" and have images, video, sound as input and output, even the text can get you pretty far. All the works of literature that write about what humans feel. The understanding of biochemistry that governs the human body will also help. Sure this kind of intelligence is a true "alien" intelligence. It is the first alien intelligence mankind contacts.

And then the last part: The argument that these systems do not have a "world" they live in where they can collect their own experiences. Now this ignores that there is reinforcement learning.

A chess computer learns by playing endless games of chess. It has a 8 by 8 checkerboard world. Soon the coding assistance software will not only "autocomplete" your code but will write programs, then compile then, run and test them, profile their performance and learn from that "experience". It has an abstract world of "software and programming" where it navigates and learns.

The "universe" of these systems is quite different from ours. So there will always be a bridge where we do not understand them and they do not understand us. But I find it quite arrogant to assume that THEY are limited where actually WE are. THEY can create new universes where they "live" that we can't.

As Wittgenstein famously said:

if a lion could speak, we could not understand him’ because a lion’s form of life is so alien to ours that we cannot seriously claim to know what the lion means by what the lion says.

So the main fallacy of the critique on AI is that they confuse "intelligence" and "human intelligence".

And all of this is not science fiction: All the above either exists or is actively worked on and literally trillions of dollars of budget are directed that way.

So the answer to the question is:

Can it ChatGPT understand text:

  • currently not but within the next few month most likely yes.
  • yes. but it will never be able to understand it the same way as a human. as much as we can not understand the lion.
2
  • The Chomsky reference is great, and the points are also great. For the conclusion, "within the next few month most likely yes ... [but not] the same way as a human". I think that LLMs need sensors and different algorithms in order to understand text (won't be LLMs anymore), and the understanding will be predicated on what we do: looking at text. But the connective relationship between storage locations is spot on. How the connections are established and followed will be different. But I think the understandings such as of "tree" and of "mud" can easily be the same as a human's.
    – Roddus
    Feb 25 at 0:03
  • But to add to this, I think the machine's sensor and effector apparatus generally will need to be much the same as ours, since understanding "mud" includes its smell, feel, taste, viscosity etc., and one could also say includes the ability to avoid slipping when moving over mud.
    – Roddus
    Feb 25 at 2:28
1

The answer is pretty simple and short:

The argument is false. Here's why:

The presence or absence of text in the guts of computing machines is completely irrelevant. What matters is information. Shannon information. That's what human language is about and what LLM models, and deep learning models in general, deal with.

You better coherently establish the meaning of understanding if you're gonna argue computers don't have it. Since I don't suppose the author successfully and satisfactorily defined it, his argument is doomed.

Not the answer you're looking for? Browse other questions tagged .