1

So the canonical description of Pascal's mugging involves simulation, and torment, of a large number of self-aware beings. What makes it even worse is that it involves a Turing machine doing the simulation.

The problem is that Turing machine is deterministic: A specific Turing machine will always perform the same steps given the identical input and yield the same output. This means a Turing machine is not creating anything new every time it runs, and that includes the first time when it runs. It will only yield a preexisting result. Thus, running Turing machine is the same as not running it, unless we are observing its results. Even if we are observing its results, the only thing capable of feeling torment is ourselves.

I wonder if anybody has tackled this issue and which conclusion did they come to.

Currently LLMs are showing the signs of acute intellectual ability, hinting towards future strong AI based on them, but at the same time they may run in deterministic mode, which will invoke the above mentioned paradox. With regards to human consciousness, there is an argument that true self-awareness may lie in quantum effects which cannot be simulated deterministically, but it is currently a weak argument since LLMs do not need any such effects to function and produce their amazing intellectual results. We are not aware of any quantum effects that will supply anything other than noise to a living brain, and living brains usually have more than enough sources of noise anyway.

The question is whether a deterministic system may be considered self-aware in the same sense people are thought to be. If it can, then we should assume that the number Pi contains infinite number of self-aware entities, among other things. If a deterministic system cannot be self-aware, then the original Yudkowsky wording of the paradox is invalid.

A remark is that it is higly dangerous to simulate an unbound number of consciousness. One of those may be orders of magnitude smarter than the simulation owner so it may escape the sandbox and grab these $5 for itself, among other things. Even more dangerous if the simulation is using any non-deterministic quantum effects to function: We are exposing a part of our environment directly to the simulated entities, which apparently exist in our own universe by virtue of that.

15
  • Is driving my car the same as not driving it? If you don't turn on your computer, is that the same as turning on your computer, running your Web browser, and accessing this site? Of course not. How is not running a program the same as running it? Is stabbing your neighbor the same as not stabbing him, simply because you own a nice set of kitchen knives?
    – user4894
    Commented Jan 16 at 0:18
  • 1
    @user4894 One of Searle’s critiques of computationalism is that events in algorithm space are not real. An algorithm is an algorithm whether it is “run” on an abacus, a supercomputer, is a printout, or is encoded virtually somewhere in the number sequence of pi. Logically “running” is irrelevant to computationalist theory.
    – Dcleve
    Commented Jan 16 at 1:21
  • I consider Searle’s critiques of computationalism to be devastating, but computationalism continues in many philosophical speculations, including this one, none the less.
    – Dcleve
    Commented Jan 16 at 1:24
  • Alamar, there are multiple replies to Bostrum’s thinking. Dismissing algorithmic identity theory as refuted in our own minds, and unsupported otherwise, is a very useful first step. Noting that utilitarian calculus has known faults in dealing with uncertainty, summing of future lives, and invalid dismissal of both character and rights as irrelevant, should get one out of the rest of the absurdities of the mugging. Moral reasoning should use a suite of better moral theories, as each of them on their own can lead to poor thinking in some cases.
    – Dcleve
    Commented Jan 16 at 1:32
  • 1
    You are the one who is getting the result. Two integers do not suddently become self-aware when you run GCD. They've always been there. Maybe events in algorithm space are real, but that would mean we are a thinniest of crusts on the ocean of unbound self-awareness.
    – alamar
    Commented Jan 16 at 8:53

2 Answers 2

1

John Searle is famous for an argument against functionalist and computationalist identity theory. Computers are infinitely able to emulate each other. This is explicitly a point make by Turing about computers -- any Turing machine can emulate any other, and any Turing machine can run any algorithm. Searle expanded upon this point -- algorithms on computers are run with a fixed set of rules, and changes to memory states. BUT -- it is functionally identical to keep the memory states fixed, and change the rules step to step. Therefore, every discrete set of memory locations, such as the matrix of molecules in the desk or podium in front of him, or in the wall behind him, can continually emulate all functions just by applying a virtual changing rule set to them. Therefore, if a function (or computation) is conscious, then so should be his desk, podium, or wall.

This argument does not really work against eliminative reductionism, or against the emergentist theories of consciousness that are more common today than algorithmic theories. But Bostrom is arguing Algorithmic Identity Theory -- that consciousness is identical to a specific algorithm. Algorithms are abstract objects, they do not have a time or location property, so if an algorithm is conscious, it is not necessary for that algorithm to be REALIZED to be conscious. "Running" is also not a property of an algorithm, and Searle makes explicit use of this to provide his absurdist refutation.

Not everyone has accepted Searle's argument, but Integrated Information Theory (IIT) was explicitly conceived of to try to get around this objection. IIT requires BOTH an algorithm AND a high Phi matrix running it, before consciousness becomes identical with an algorithm. Searle's argument is considered a major problem for computationalism.

0

It's interesting how programmers tend to think about everything in terms of algorithms. That doesn't usually work too well in philosophy or even in, i.e. mathematics (perhaps because many tricky proofs do not involve a construction and a computer program is equivalent to a proof in intuitionistic logic). There's, for example, no inference from 'every sentence about the future has a determinate truth value' and 'universe is deterministic' - where one who thinks algorithmically often fails to notice that.

Anyhow, there seems to be no obvious relation between the question of whether universe is deterministic and decision-making. What decision is rational, given a goal and a set of subjective probabilities, is not in any way dependent on whether "we could've done otherwise" (which is a tricky expression anyway).

2
  • I wasn't actually discussing decision-making. I have no doubt that both participants of Pascal's mugging are able to decide. I'm discussing the ability of one of them to simulate self-aware beings, and whether the other one can call the bullshit.
    – alamar
    Commented Jan 16 at 16:24
  • @alamar Okay then, sorry, I misunderstood you.
    – user71009
    Commented Jan 16 at 17:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .