Can Consciousness be Computationally Reconstructed from a Painting?

Question

Suppose that Pablo Picasso was undergoing an arbitrary conscious experience while putting the finishing touches on Guernica. Would it be possible to reverse-engineer that conscious experience computationally, merely by analysing the finished painting pixel-by-pixel?

For the sake of argument, let's assume that consciousness is, somehow, computable. Then, assume that your state of consciousness is in some meaningful way transferred onto an artwork when you create it. Whether it's an oil painting or a video game or the complete works of Shakespeare, assume that the state of your mind is imprinted onto the given canvas/game/text substrate at the moment of creation. Would it then be possible to extract that state of mind by running a pixelized version of the artwork through some kind of Oracle LLM AI?

After all, it should be possible to reverse-engineer software from hardware. Given a MOS 6502 Microprocessor, one could analyse the flow of electrons across transistors and formulate a meaningful interpretation of a Donkey Kong video game running in software on that hardware.

Isn't this essentially the same problem?

Are all artworks essentially a cryptographic zipfile of the artist's inner life?

Answer 1

This computational framework is clearly unnecessary and completly unenlightening. You are saying: Can we predict some range of internal, conscious states of a subject based on some of their responses to stimuli? The answer is: we probably don't know because it's a question which requires very careful empirical studies and the way you formulated it is too imprecise. Clearly, even if you make it precise enough, it might be highly dependent on the person. We don't know.

Answer 2

This isn't possible with current technologies, so it's best viewed as a thought experiment about the ramifications of the common assumption that consciousness reduces to brain-state.

You have a second assumption here, one that's less common--that the physical output of a person's life is a de facto recording of their inner state, whether or not we can interpret it. While it this seems farfetched, I think I can perceive your reasoning. First, we know that, for instance, sounds can be reproduced from the markings made by a needle--the basis of phonograph recording technologies. We also know that sounds previously not recoverable from primitive recording techniques have been reconstructed by advanced computers. Finally, we know that simulations of--for instance--a particular artist's painting style can be reproduced by feeding a modern AI examples of the artist's paintings.

Modern AI is a bit of a "black box," so we don't know much about what goes on inside of it. Most people would assume the paintings are just being imitated in a more mechanical way, but some people might hypothesize that the artist themselves, and their creative spirit, is being in some way simulated internally in order to create the new paintings. This is all too speculative to be decidable, but it does serve as an interesting exploration of the implications of certain answers to questions about consciousness and identity that currently remain open.

Answer 3

I'd agree with @abcga that the whole computational angle is mostly irrelevant to the question.

I mean first of all the "pixel-by-pixel" thing doesn't make any sense. I mean I'm not a great artist myself, in fact my drawing abilities might be dwarfed by pre-school children but even I know that Picasso didn't draw in pixels. So by adding those you'd add more questions like which resolution? If you make it too high, that becomes intractable if you make it too low you lose information encoded in the image.

Also conceptually isn't that what at least some art critics and observers try to do already? Like idk some people think they can tell about eye diseases of the painter by features of the image or a slight shadow on paintings could be emblematic of air pollution in that time or stuff like that. Light and shadow could encode sympathy or antipathy, distorted proportions could imply significance or a lack of talent. And often enough the artist "hides" (often it's also plainly obvious or at least they think it is), a message in their art.

So the idea is not completely far fetched. However similar to most compressed or encrypted messages, there's not just one way to read them, but next to infinitely many. In the end only the artist or the person for whom the message was originally recorded can tell. So you would need to reconstruct that person to decode the message and you want to do that by decoding the message? That sounds rather difficult...

And a computationally heavy machine isn't going to simplify that much. It could sift through the information faster (in theory) while in practice it would need to learn from the ground up what it means to be human in general and what it means to be that artist in particular, so unless it's something trivial (which you could have seen without the machine) it probably will be in the domain of uncomputeable in reasonable time.

Or if you manage to do so it will come with technological advancements and philosophical implications more interesting than this question. Though that's pure speculation.

Also just because the artist encapsulates some of their feelings, emotions and conscious ideas in their art, doesn't mean it's a full transcript of their consciousness, like if a future civilization finds your shopping list they won't be able to reconstruct who you are as a person despite the fact that there is some information about you on that piece of paper.

Answer 4

Your thought experiment involving computation invokes some philosophical topics. In such a philosophical problem, the point is not to answer the question. But to take stock of the questions it raises. This type of meaning is known as inquisitive semantics to some. Your question certainly raises at least 3 thorny philosophical considerations. For instance, you may be committing a category mistake by placing CMOS devices in the same categories as brains isolated or embodied in consideration.

Brains Are More than Turing Machines

(Consider the PhilSE question Why do some physicalists use the Turing Machine as a model of the brain?. Also Consciousness in Simulation theory & AI, why do some believe that it is even possible?.)

For instance, when you talk about reverse engineering a MOS 6502 and reverse engineering the human brain, you are in a different complexity class. For instance, computer hardware is amenable to description by formal semantics. Consider the use of VHDL. From WP:

The VHSIC Hardware Description Language (VHDL) is a hardware description language (HDL) that can model the behavior and structure of digital systems at multiple levels of abstraction, ranging from the system level down to that of logic gates, for design entry, documentation, and verification purposes. Since 1987, VHDL has been standardized by the Institute of Electrical and Electronics Engineers (IEEE) as IEEE Std 1076;

Can the same claim be made about the human brain? While I'm aware of no canonical disproof, notions of cognitive architecture and embodied intelligence (SEP) are very complex indeed.

Analog vs. Digital

(Consider PhilSE question Is it there any specific and well known continous/analog alternative to Wheeler's discrete "It from Bit"?. Also Digital Minds and their perception.)

In systems that perform physical computation (SEP), it is clear that modeling Guernica with pixels is not a faithful reproduction. Pixels are mental objects, something many would consider abstract objects (SEP). But pixels are an approximation of the physical objects they represent. Consider the difference between a picture of a tin of cookies and the actual tin of cookies. Reproducing the picture is an easy problem for computation. Photographic copiers replicate pictures within a specification, for instance, differences being contained to not capable of being detected by the naked eye. But replicating a tin of cookies? The brain is capable where the copy machine is not.

Non-Determinism and Underdetermination

(Consider the PhilSE question Is there a term for materialistic non-determinism?.)

Part of your question asks about recreating a physical object. The ability to recreate a physical object would be a deterministic process. Notice how standards suddenly come into play. Say you accept a standard for physically replicating Guernica. Forgers are quite good at recreating faux art. But does that mean that if one forger can fool art experts that the forger suddenly has insight into what Pablo Picasso thought about his breakfast preferences? Human brains may be able to mimic other brains, but that mimicry doesn't confer ALL properties of the brain it is mimicking. Clearly, just enough to replicate one very specific skill at most.

In this way, the universe is non-deterministic. Unlike a math proof that allows you to move back and forth between steps with impunity, physical machines and their states are in some sense, lossy. Thermodynamics, for instance, means that the energy of the system inevitably is noisy. Systems of physical computation are not identical to the abstractions we create from them. Computer software has bugs, computer hardware has failures, the human mind forgets, etc.

Conclusion

Your Gedankspiel raises some fundamental questions. You ask:

Can Consciousness be Computationally Reconstructed from a Painting?

Probably not any more than photocopying a book allows you to rebuild the author from scratch. However, it raises some excellent questions about determinism, analog and digital reproduction, and physical computation. Clearly reverse engineering a microchip is not the same as rebuilding (if it's even possible) a consciousness mind. They are very distinct classes of ontological commitment.

Answer 5

The ideas and associations of a painter cannot be reconstructed from the pixels of his paintings:

1. Because the conscious mental processes of the painter are dynamic processes, while the pixels on the canvas are a frozen, static state.

   Hence already on this level, there is much more information contained in the conscious process than in the pixel arrangement.

2. More important, I consider the level of conscious perception of ideas and associations as quite different from the level of pixel arrangement.

   Even when considering the painting of a painter as a whole and analyzing it with the means of art history - biography, time, social context ... - it is often not possible to extract with certainty the intention, the artist had with his artwork.

Answer 6

From a computational perspective, if you make the first two assumptions, the only real issue is whether the transference is a one-to-one or one-to-many mapping from experience to artwork. If it is many-to-one (i.e. multiple experiences are mapped to the same artwork) then you will not be able to extract a unique experience. Unfortunately, if the analysis is "pixel-by-pixel" you have almost certainly made it a many-to-one mapping as the spatial (and probably intensity/hue) quantisation involved will eliminate the subtle differences between similar experiences.

TL;DR what @TKoL wrote (+1)

Answer 7

Yes, it can be reconstructed in theory because the consciousness itself is computational. We can see why that is true once we understand the nature of consciousness itself.

We know that Plato and Greeks in general (quoting Heidegger) understood that genuine knowledge is visual. That's why no one can be told what the Matrix is, that's why people cannot comprehend the logos even after they've been told about it (that's Heraclitus), that's why we often have hard time understanding ancient texts. What is communicated verbally, the words we read or hear -- that is not knowledge. Rather, words (and language in general) is an imprecise way to describe the real thing -- the real knowledge, which we would have to reconstruct from the words and, again, visualize in our minds.¹

Without this capacity to visualize things in advance (a priori), not only we would be deaf, but also blind -- we will see shapes and colors, but we won't be able to recognize the things we are looking at.

And it is this capacity for knowledge that also makes us self-aware and conscious -- we understand/know our world inasmuch as we can visualize the simulation of it (the logos), and we understand/know ourselves inasmuch as we can visualize a model of ourselves as part of the larger simulation of this world... which brings us to your question.

Picasso had to picture Guernica in his mind before he put it on the canvas. That capacity -- our brain's capacity to visualize a model of the world -- is indeed no different than the capacity of MOS 6502 to visualize Donkey Kong. The architecture of our prefrontal and visual cortex is quite different from that of a CPU/GPU combo, but functionally they are doing the same job. Therefore, at least in theory, everything that we are, our knowledge and our consciousness can be reconstructed from the electrical currents in our brains.

Or, in other words, we are the information that is being continuously processed, stored, and rearranged inside our bodies. To what end? -- perhaps, for a human individual, the meaning of life is to assemble, in their mind, the most complete and accurate simulation of the reality. And to become happy -- but good luck finding that one without a map.

¹ And no, the success of that reconstruction is far from guaranteed -- but it is crucial to the success of communication as a whole.

Answer 8

You might deduce something, but it won't be much other than the obvious, and any will probably be very subjective.

Let's try something easier. Supposing you wrote something. Does your handwriting tell anyone anything about you? There are generalities. If you have small, neat handwriting then your handwriting is small and neat, which may extend to other characteristics. If your handwriting is large and expressive, then you may be the same. But the details people claimed for graphology, such as leaving the tops of your 'a's open implies dishonesty have little basis. And sometimes, your handwriting may be like the original John Hancock's signature, which says 'I like the way this looks.'

We do have a handwriting style. If we write with a pen, or a piece of chalk on a blackboard, or an aerosol can on a wall or an engraving tool on metal under a microscope, the speed of writing may vary but the style remains much the same. We seem to have some kinematic model of writing which we can transfer from one set of muscles to another (hand for pen, arm and shoulder for aerosol, fingertips for engraver). This gives us a hint about how we may think about writing (and painting) but it may not tell us much about our mood at the time. And what we are writing may have no connection with the style if we are copying something.

Answer 9

Define consciousness sufficiently, and you can get an answer. Unfortunately there is insufficient consensus.

One can comfortably say that given a painting, or even an entire collection of paintings, it provides information about the painter. We may even quantify that information in terms of bits, providing a mathematical object to capture the information in the painting.

Such a bit string is finite. If we consider infinite strings of information, the story might change, but I think it's hard to argue for that. And its certainly hard to argue if you thought in terms of "pixel by pixel." This means the information strings can, at most, capture a countably infinite number of classes of consciousness that might produce the painting. Likely it ends up being a finite number of classes, but at the theoretical best, it's a countably infinite number.

It is not known whether there are an uncountably infinite number of possible consciousnesses. In the end, we don't know, but I do get an informal feeling that we consider consciousness to be "bigger" than a simple bit string.

If this is true, then we can state that analyzing a painting cannot be reconstructed from a painting. At best we could re-create a consciousness from the class of consciousnesses capable of producing that painting, but its very easy to argue said class is infinitely large.

Answer 10

"Can we reconstruct the pig from the sausage?"

There are many ways in which we can reconstruct what the pig would have looked like, how the pig came into existence, and how the pig was turned into sausage. With suitable Jurassic-Park-style technology we could perhaps even recreate something that's superficially similar to the original pig.

But we can't put that pig back together again and have it run around squealing. It's a one-way process.

You can put whatever artistic fluff you like around the question, but it remains the same question, and the same answer.

Answer 11

To use your Donkey Kong analogy, would you be able to reconstruct the software from a screen shot of the game? Even assuming that you know how the hardware works, the answer is no. The same visual image could be created by any amount of different programs, only one of them being the actual Donkey Kong game.

The best you could do is to use all of the works of an artist to construct something that generates similar art. I.e. you could create an AI that emulates the art style of the artist. That would tell you nothing about how the artist would behave in some non-art related situation, or even whether or how his art style might change over time.

Answer 12

I could hardly be more confident that the answer to your question is no. You are assuming there is a unique, complete and computable correlation between one's mental state and the appearance of the results of one's handiwork. I can readily think of at least ten reasons for considering the suggestion implausible. For instance, let's suppose I, an abstract artist of international notoriety, create a painting which consists of a horizontal black line across a white canvas. What information do you suppose you might extract from that which would allow you to recreate my mental state? Moreover, suppose that having contemplated my new artwork for several weeks, I decide it should really be a diptych, so I paint another canvas indistinguishable from the first. According to your scheme, my mental state when painting the second picture should be indistinguishable from my mental state when painting the first. Either that, or you consider the two paintings collectively to be the artwork, in which case your computer program needs to account for my mental state throughout the period between the painting of the two halves of the diptych. Or let's suppose that some way through the painting of the second horizontal line, the sunlight through the north facing window of my Paris garret catches the canvas in a way that triggers a moment's intense melancholy as I remember my short lived affair with the Spanish post-impressionist Griselda Terrini La Luna- how does your computer program infer that from the pixelated image? Etc etc etc etc etc.