Devising mathematical or quantitative theories of meaning

Question

Around 1948, the mathematician and electrical engineer Claude E. Shannon presented work that would eventually lead to information theory. A mathematical theory based on uncertainty and probability, which allowed to quantify the "amount of information" contained in a signal or a set of discrete data.

Paradoxically, given a sequence of length N formed by signs of some alphabet, the sequence contains the most information when it is "random", i.e. there is no redundancy or predictability within it (note that no strictly finite sequence is random, although many finite sequences can be described as quasi-random). On the other hand, fully predictable sequences such as "AAAAA ... AAA" have minimal information. Thus, it is curious that both minimal and maximally informative sequences are meaningless. The same happens with digitized photographs, a monochrome white (or black) background has minimal information and null beauty, an image of the same size with randomly assigned colors and brightness has maximum information, but humanly it has not much meaning. A beautiful photograph of the same size in pixels has an intermediate amount of information between maximum and minimum, but has much more meaning or interest than the extremes.

All that leads one to think, what a quantitative theory of meaning could be based on, in particular:

1. Is meaning an intrinsic property of objects (as is Shannon's quantity of information) or is it something entirely subjective that therefore does not admit of objective quantification.
2. And, if we somehow restrict the possible images or sequences, then for that set a quantitative measure of its meaning could be constructed.

Answer 1

I assume the thrust of your question is that quantity of Shannon information doesn't line up with our informal idea of quantity of information, and you are wondering if there is some formalism that does a better job of mapping to our intuitions.

First, realize that Shannon information is not what anyone meant by "information" before Shannon's theory, and it's not what the informal word "information" means now. The same thing holds true of many technical terms like "energy" in physics, "market" in economics, "run" in baseball, etc.

It is usually a mistake to take one of these technical terms and try to somehow unify them with the informal concept with the same name. For example,

A walk is when you are in no hurry because you can't be thrown out, so you can just amble over to first base, but what about a home run? The ball is not in play, so you can walk on a home run. Isn't that paradoxical?

No, this isn't paradoxical, it's just the peculiarities of natural language. Why do you park on a driveway but drive on a parkway? Once you realize that Shannon information is only allegorically related to informal information, it becomes clear that the peculiarities you have noted are amusing, but not significant.

There has been a lot of mathematical work on meaning. The field that studies meaning is called semantics. Semantics is studied in at least three contexts that I know of: logic, linguistics, and computer science.

In logic, formal semantics is restricted to the study of statements. In this subfield meanings are generally represented as propositions. The theory is very formalized and mathematical, but I don't know of any work on quantity of meaning.

In linguistics, there is a lot more to study, and semantics turns out to be a lot more complex, so it is not always handled formally. There are even theories in linguistics that meaning is a poor way to think of how communication works in general, and that there should be other ways of understanding communication.

In the field of programming-language semantics, programming languages are mapped into some formalism; perhaps a simpler language, or mathematical functions, or operations on a virtual machine of some sort. Again, I'm not familiar with any work on quantity, although there is some work on complexity of programs, which might amount to the same thing.

Once you have a formalization of meaning, you might be able to quantify it in some way, but you are always going to run into difficulties because meaning is fungible. Does "There are irrational square roots" have more information than "There are flying mammals"? In order to explain "irrational", you have to use a lot of words, but "flying" seems pretty simple. However, that's just because we grow up seeing birds and insects, and we don't grow up dealing with irrational numbers. The notion of flight is more complex if you try to explain it to someone who doesn't have the same background (say an alien species that lives underwater and doesn't even know what air is).

Answer 2

Since you asked for mathematical concepts of meaning, let's begin by considering a logical proposition P in some axiomatic system, such as ZFC. What is the meaning of P?

Now, one perspective is that we would first map the terms and symbols of P, to some other domain, a model. And the meaning of P would be given by what it says about objects in the model.

But this raises questions. For one thing, how does the axiomatic system relate to the model? The association with the model is something extra, outside the axiomatic system itself. With ZFC it may not even be possible to specify in words which model we are speaking of. Some of these problems are discussed here.

For another thing, it just kind of kicks the can down the road; instead of talking about the meaning of propositions in the axiomatic system, we are now talking about the meaning of objects in the model. If the first wasn't clear, the second might not be clear either.

So let's set that aside for now. Instead of mapping to a model, let's instead talk about meaning intrinsic to the axiomatic system. By this I mean, we want to know the role that P plays, in relation to other propositions in ZFC.

To momentarily switch gears, using an analogy to the English language, we may consider the role of the word "cat." The role of the word "cat" would be partly given by when it is appropriate to use that word, depending on what other words surround it and what is happening in the physical world that might cause us to talk about a cat. These are the conditions that motivate the speaker to say "cat." This relates to the intended meaning of the word. The other part of the role of the word "cat" would be given by what the listener can learn from the fact that the speaker has said "cat." The received meaning of the word.

Similarly, the role that a proposition P plays in its axiomatic system can be generally divided into two parts:

• In what circumstances is it appropriate for us to infer P? We may ask when we can produce P in a single inferential step, given other propositions. We may also ask about chains of multiple inferential steps that allow us to eventually produce P from other propositions.
• If we are given P, what can we infer from it? We may ask about what we can derive from P in a single inferential step, in combination with other propositions. We may also ask about what we can derive from P in a chain of multiple inferential steps, in combination with other propositions.

The first may be called the "upstream" meaning: the circumstances that allow P to arise. And the second may be called the "downstream" meaning: the circumstances that P allows to arise.

Going back to the "cat" analogy, the upstream meaning is like the intended meaning of the word "cat," and the downstream meaning is like the received meaning of the word "cat."

upstream meaning = circumstances under which a sign arises = intended meaning

downstream meaning = circumstances that a sign results in = received meaning

We may see a parallel here between introduction rules (upstream meaning) and elimination rules (downstream meaning) for logical symbols.

We're being a bit vague here, though, when we talk about "circumstances." This is not mathematically precise. In the case of a formal axiomatic system, we may make it precise. We can say that the upstream meaning of P is identified with the set of all proofs having P as the conclusion, and that the downstream meaning of P is identified with the set of all proofs having P as the first premise.

Answer 3

I think you're conflating information content and meaning, which are actually two entirely different ideas.

When you say, the "sequence contains the most information when it is random", you're quantitatively talking about Shannon entropy, or pretty much equivalently, Kolmogorov complexity, https://en.wikipedia.org/wiki/Kolmogorov_complexity. Very briefly, given a string, its Kolmogorov complexity is the length of the shortest computer program that can output that string. If it's random, then there's no algorithm that can generate it, and the program simply has to store the string and print it. There's no shorter program than the string itself.

But all that has nothing to do with meaning, which is semantics. And there are three quantitative theories of semantics: operational semantics, https://en.wikipedia.org/wiki/Operational_semantics, algebraic (or axiomatic) semantics, https://en.wikipedia.org/wiki/Algebraic_semantics_(computer_science), and denotational semantics, https://en.wikipedia.org/wiki/Denotational_semantics.

And I'd guess it's denotational semantics that's closest to what you (and most people) typically have in mind when asking about (your words...) "a quantitative theory of meaning". In this case, the theory involves a formal grammar, https://en.wikipedia.org/wiki/Formal_grammar, and a semantic function that maps syntax (well-formed formulas as per the grammar) to semantics (the meanings of wffs in some domain/set of meanings).

The grammar for denotational semantics is typically BNF (Backus-Naur form), https://en.wikipedia.org/wiki/Backus%E2%80%93Naur_form, which can be illustrated with the simple example of numerals,

<digit> ::= 0 | 1 | 2 | ... | 9
<numeral> ::= <digit> | <numeral><digit>

which I've simplified even a bit more than usual, and I'm sure you get the general idea (exercise for reader: add a few more rules to prohibit numerals like "00000123" with arbitrary strings of leading zeroes).

And now the meaning of wffs in our numeral grammar is obvious, though I'll illustrate its formal semantic function below. The important point is that a random string wff like "8430927" has just as much meaning as "1234567", i.e., they're both integers. In particular, their Shannon entropy (information content) has nothing at all to do with their semantics (meaning), or lack of meaning.

P.S. Just for completeness, the semantics of a wff is usually denoted by [[wff]], and then for our numerals

[["0"]] = 0,  [["1"]] = 1, ..., [["9"]] = 9
[[<numeral><digit>]] = 10 * [[<numeral>]] + [[<digit>]]

Denotational semantics is usually used to specify the meanings of programs written in various and sundry programming languages. And, of course, the BNF grammar for such languages is lots more complicated than our numeral grammar above. But such grammars can always be constructed. And then the corresponding semantic functions are also lots more complicated: the meanings of programs are in the domain (the space) of functions from integers to integers, i.e., a program takes input as an integer, and generates output as an integer. The extent to which there may be a BNF grammar, and semantic function to some domain of meanings, for English is open to question.

Answer 4

Short Answer

It's not true that random sequences of letters necessarily do not have meaning. I need simply produce a simple counterexample. If meaning is understood as association between sequences of graphemes and human experience, the simplicity of the binary representation that encodes the meaning is in no way tied to meaning. That being said, you can create any theory you want. Ultimately, you'll need a community to read, argue over, and adopt your theory as an explanatory tool.

Long Answer

One can design a cipher so that one maps meaning indirectly to a set S of n randomly generated sequences. Now, let's say the set looks like:

ant asleep at the wheel = alsegn
bee flying at a frightening pace through the world = febllp
cat sitting on a really, really, really hot tin roof = quzifes
all words in the OED starting with a letter = ggmoxf
...

Now, each random sequence of has cryptographic utility as well as meaning. Thus, the mapping of meaning to, let's say binary sequences that are used to encode the alphabet of any collection of graphemes has no relationship of proportionality between length and the complexity of meaning.

But let's roll your idea back to just standard English vocabulary. What does it even mean for meaning to be quantified? Does semantics even give us any leads? Do philosophers of language have anything corresponds? If one takes a pictures of something meaningful to a human, are the configuration of pixels somehow indicative of the complexity of the image? How about the meaning of the picture? Well, there is the principle of compositionality. From WP

The principle of compositionality states that in a meaningful sentence, if the lexical parts are taken out of the sentence, what remains will be the rules of composition. Take, for example, the sentence "Socrates was a man". Once the meaningful lexical items are taken away—"Socrates" and "man"—what is left is the pseudo-sentence, "S was a M". The task becomes a matter of describing what the connection is between S and M.

So, certainly in natural language, the complexity of meaning seems to enjoy a "rough" proportionality between morphemes and amount of meaning. For instance, "myocardial infarction" seems to have more meaning than "heart" intuitively. And "I suffered from myocardial infraction last day of last year" has even more meaning there. But in some ways, it counts on how you measure meaning. For instance, can you say with certainty that "borborygmus" has more meaning than "fart" despite one has more syllables than the other? Is their conceptual semantic content not almost identical? In fact, besides multiple theories of semantics, philosophy recognizes that meaning has all sorts of dimensions. Which is meaning? Again from WP:

The major contemporary positions of meaning come under the following partial definitions of meaning:

Psychological theories, involving notions of thought, intention, or understanding;
Logical theories, involving notions such as intension, cognitive content, or sense, along with extension, reference, or denotation;
Message, content, information, or communication;
Truth conditions;
Usage, and the instructions for usage; and Measurement, computation, or operation.

In conclusion, Shannon's theory is a great leap forward in quantifying strings of symbols, and has philosophical import in measuring simplicity and complexity, doing hashes for algorithmic efficiency, doing cryptographic work, etc., but to impute some sort of essence on physically real objects based on their natural language representations is dubious for reasons not the least of which that strictly speaking syllabaries are artifacts of human language communities, not aspects of objects. Hence, to do so would confuse the territory for the map; in other words, would mistake the symbol for the referent.

So, from a naturalized epistemology, the theory of information is best understood as a mathematical tool for analyzing languages or sign-systems invented by people, where the latter are abstractions.

Is meaning an intrinsic property of objects (as is Shannon's quantity of information) or is it something entirely subjective that therefore does not admit of objective quantification.

Meaning is probably best understood simply as an experience of agents and might be understood in terms of mental representations.

And, if we somehow restrict the possible images or sequences, then for that set a quantitative measure of its meaning could be constructed.

Sure, one certainly can measure the language used to convey meaning, but is that the same as measuring meaning itself? Perhaps with fMRIs measuring NCCs?

On the whole, you're free to create whatever theory of meaning pleases you. But ultimately such a theory will be judged on its explanatory power or utility by your fellow thinker, and there's a fine line between crank and genius.

Answer 5

I would differntiate information, as being impersonal, from knowledge and meaning, as involving situating yourself to information. Discussed here: Language, thought, cognition (we use intersubjective ideas to generalise the self being situated).

This area of concern about the transition between chaos and order, is Complexity Theory, the study of non-linear dynamics (which includes 'chaos theory'). A signal being sent down a Shannon channel can deteriorate, but is very unlikely to reconstitute the original information, and this has been proven to be formally equivalent to the second law of thermodynamics. The principle of Conservation of Information, theorised but not proven, would indicate signal loss is really spreading-out or diffusion of information, and it remains in the system as a whole, but typically lost to the most diffused energy modes such as thermal excitations. 'Amount of information' is always relative to the original signal - and similarly, we cannot measure entropy as an absolute value, but only entropy change (there may be 'hidden' microstates). I make the case in this answer that we should regard 'true' in a similar fashion, in relation to correspondance with expectations: Why is a measured true value “TRUE”?

Some philosophical approaches that relate to the progressive developments in information structures in philosophy, that spring to mind:

• Baudrillard's degrees of abstraction of simulacra
• Kuhn's paradigm shifts
• Vervaeke's salience landscapes
• Hegel's dialectic

I find it interesting that often in philosophy we discuss information and meaning, but rarely the quality of how it seems to be getting compounded, and to be getting integrated into more powerful and versatile structures - which seems directly at odds with the general tendency towards increasing entropy. It has struck me that Baudrillard and Kuhn don't deal with how the memetic structures they describe, are in a process of natural selection, which is causing this. Language pictured as organising salience landscapes, can help us understand that it these tend to become more refined and versatile over time, not the reverse. The dialectic seems a reasonable model, but a bit woolly and apt to be narrative post hoc rather than any use for prediction.

Integrated Information Theory attempts to describe more and less powerful information, and make a picture that can account for different degrees of awakeness and consciousness. There's been little progress with it as an approach, and a lot of scepticism how useful it is within consciousness studies.

Deutsch & Marletto's Universal Constructor Theory could I think point towards a generalisation of theory of computation, that can account for this quality of progressive developments in information. It is notable that it very naturally accounts for Maxwell's Demon, by being concerned with what is involved with a subsystem returning to it's original starting state, including wiping any memory storage (the entropy consequences of that being the main accepted way to dismiss Maxwell's Demon violating thermodynamics). It explicitly attempts to account for how random systems can become information-holding ones.

We were recently discussing on here frequency analysis of languages in relation to evaluating dolphin signals. There is substantial dispute by scholars about how good a guide it can be to the information in a signal, because other reasons can cause these patterns - they indicate complexity, but can't distinguish that from non-mind causes that could quite reasonably be involved. Turbulence is complex, that doesn't mean every example of it is a message.

Direct answers:

1. Information is relative in Shannon's theory not abolute, measured between input and output

2. If we could construct a quantative theory of meaning, talking to dolphins would be easy. I argue here that a 'Rosetta stone' equivalent is required for translation, that shared experience of the modes-of-life of multiple languages is needed by one individual to act as translator: According to the major theories of concepts, where do meanings come from?