In an answer to this question I made the rather bold speculation that if the Turing Test is taken to be our defining criterion of a consciousness of human level, then all questions in the ethics of simulated minds can be reduced to human equivalents with a canonical reformulation.

To put it another way:

Mary and Siri live in TurEngland, a place where the laws are set by a collective of simulated minds, operating on an empirically validated utilitarian calculus- every 'mind that passes a turing test is considered in the calculations, and the calculus 'pulls back functorially over the criterion' meaning all measures of Eudaimonia are demonstrable in mental states recognised as such by a Turing Test.

Siri is a campaigner for 'mind supremacy, with many of his eudaimonia states skew-distributed toward a world where humans counted less. Mary has the opposite skew against that particular variable and wishes for a world where humans rule the roost. Can either devise a scenario in which their hopes are realised? Can they find a way to break the system altogether?

To express the question in its broadest terms:

If one substitutes 'prescriptivist processing network' or 'Categorical imperative evaluation- thingy' for 'utilitarian calculus', one has, I think, still an interesting question. That is:

"Which, if any, of our current moral systems fail to provide answers, or 'break' in some way when Turing-Test-passing minds are inducted into moral personhood?" (and which are therefore not compatible with the Turing world-view?)

Can simulated minds become utility monsters? Are there thought experiments set in such a world where our present ethical apparatus fails to provide an answer?

To clarify: I am not looking for an evaluation of the Turing test as such a criterion, rather to see if there are situations in which, if it is assumed to be our criterion for moral personhood, our current moral frameworks are shown to be inconsistent or incomplete.

  • Interesting. After the edit it sounds more like you are asking whether simulated minds would be (could potentially become) de facto utility monsters -- is that about right?
    – Joseph Weissman
    Mar 17, 2012 at 23:17
  • That's kind of it, will edit again now, maybe tidy it up in the morning... Mar 18, 2012 at 0:09
  • 1
    There's a lot of interesting talk on issues like this over at LessWrong
    – Seamus
    Mar 19, 2012 at 14:43
  • 2
    The Turing test had better not be a necessary condition for personhood! When does a human child become capable of passing the Turing test? Much later than it becomes a moral person, surely.
    – Seamus
    Mar 19, 2012 at 15:45
  • @seamus: good point Apr 20, 2012 at 11:19

3 Answers 3


As far as thought experiments go, the Chinese Room seems applicable. The principal implication of accepting Searle's hypothesis is basically that we accept "substrate matters" with respect to consciousness. Regardless of the complexity of the program, it cannot be considered conscious.

Chalmer's p-zombies might also be helpful in sorting out some of these issues; as Michael formulates it in his answer, if Siri x.0 is claiming it's sentient and deserves rights, on what basis should we afford these rights to it? Should we simply presume that because it passes a Turing exam that the system under examination is conscious?

Now I certainly appreciate that there's not a particularly convincing alternative means of examination insofar as sentience is considered an emergent property, not "detectable" by pure analysis of the components of the system, but rather a kind of "distributed" property arising from the integration of the components in a particular configuration: qualia are a "run-time" property, one perhaps capable of a kind of ambiguous exhibition through discourse.

A successful Turing trial, in other words, may indicate that qualia are present in the system under analysis, but on the other hand, given at least the theoretical possibility of p-zombies (in particular, an information system designed to "trick" users into believing in its sentience) it would seem difficult to state this with absolute metaphysical certainty.

Of course, it would seem difficult to state with such absolute certainty your own or others' sentience, so to some degree I certainly concede this discussion is moot.

Just to quickly restate your question, which on closer reading seems to me to be about the problems that might stem from a mixed society of biological as well as non- or partly-biological sentients. We further seem to be stipulating hypothetically that formal calculi of happiness/fun/joy can be constructed through analysis of the affective-cognitive processes of our mental Republic's members.

You posed an example about two hypothetical members of this society; I'll rename them Max and Minnie. Max's fun-vectors are optimized when non-pure biological sentients are expelled from the society; Minnie's when pure biological sentients are expelled. These would seem to straightforwardly resolve to a kind of nullity, but perhaps if the vectors were weighted slightly differently, and one or the other group significantly outnumbered the other, and their desires were relatively homogeneous on this point -- the desired expulsion (or whatever) might indeed occur. This sort of danger would appear to be generally applicable to any globally-optimizing utilitarianism; though one might perhaps hope that members of a post-singularity society might demonstrate more respect for others' differences than our societies have historically.

  • What does the having of qualia have to do with moral personhood?
    – Seamus
    Mar 19, 2012 at 14:41
  • @Seamus my initial reading of the question was perhaps not as careful as it could have been; it originally seemed to me to be (speculatively) equating sentience or moral personhood with passing a Turing test, that is, the possession of qualia and the ability to demonstrate this through discourse
    – Joseph Weissman
    Mar 19, 2012 at 16:31

None of the ethical systems will necessarily break down, since you can just treat a Turing test passing system as human. The problem is that the utilitarian calculi are necessarily ill defined.

In order to define a utilitarian calculus, you must define a sum of utility among different agents. Such a sum is not easy to define. In order for an agent to make consistent choices between probabilistic options ( a choice between a car, or 10% probability of a gold watch/90% probability for a new house) the agent must have a real-valued utility function on all options, whose values for probabilistic options are the weighted averages of the utilities. This is the Von-Neumann Morgenstern utility theorem.

The utility theorem applies for agents that can make choices, but it does not apply to two agents. Therefore while it is reasonably easy to say "I associate 3 units of my own utility to the watch, and 5 to the house" (up to corrections for human non-rational behavior). It is very difficult to say that the sum of the utility of myself and John regarding a new house for both of us is 10 units.

Since utility just does not sum between agents, it is difficult to formulate a mathematically precise version of utilitarianism.

However, Douglas Hofstadter has solved this problem with regards to symmetric games. If two agents are playing a prisoner's dilemma where their utility weight orders are symmetrical, you can naturally assume that the strategy that one prefers is also the strategy that the other prefers. If you assume they both use the same strategy, and further, that they both assume that they will use the same strategy before they pick that strategy. This implies that they will both cooperate in a prisoner's dilemma, since if the universal strategy is to cooperate, they will both cooperate, while if the universal strategy is to defect, they will both defect. Since cooperation is better for either of them in terms of expected outcome, they will both cooperate.

Two agents that do such a calculation are called "superrational" by Hofstadter. A superrational agent is one who takes into account the fact that all other superrational agents follow the same strategy. Such agents can only have one answer to any game, an answer which is only determined by knowing it must be unique ahead of time.

Since the superrational agents are following a single strategy, which is determined by all the utilities of all involved players, it is consistent to imagine an entity whose utility is maximized by the choices involved in this single strategy. The superrational strategy shared by all the players will maximize the utility of the agent formed by these players, and the agency will be universal, in that all players who are superrational will play according to this strategy. Such an agent is super-smart, and encodes the ethical choices for all possible situations. I will call this agent the monotheistic God.

The proper statement of a categorical imperative, if it were precise, is that all players should play according to the universal strategy. This is essentially the "slave morality" of monotheistic religions such as Christianity, reduced to their positivistic essentials, and not postulating any supernatural agency.

As far as I can see, beyond Hofstadter's symmetric superrationality, this idea is not clearly stated anywhere in the literature. That doesn't mean it's wrong, it just means its mine.

  • So, we could recognize an super-human AI by the fact that it makes better decisions. Including the possibility of ending its own existence, if that is what is best for all concerned (as humans sometimes do).
    – Scott Rowe
    Apr 27, 2020 at 16:41

Is this bold speculation accurate?

I wouldn't think so. It all comes down to a question of moral personhood, I suppose.

You ask us to stipulate that the Turing Test is taken to be a defining criterion of consciousness at a human level, but there's absolutely no reason to think this should be the case-- why should we assume that a piece of software capable of convincingly simulating human conversation is actually conscious?

Even if we do stipulate this, we still have no reason to think that the software is sentient in any strong sense; can it suffer, or feel pain?

If one extrapolates from Apple's "Siri", it's not hard to imagine a piece of software, a few years down the road, that can carry on an extended conversation in a convincing manner. That's not going to make rebooting my phone equivalent to murder.

EDIT: Since the question has been edited, I'll add a bit more explanation.

My argument here is a reductio-- if we stipulate that passing the Turing Test is sufficient to constitute moral personhood, all kinds of undesirable (and to me, absurd) results abound; therefore, the Turing Test is not appropriate for these purposes.

In terms of the question of how our current moral frameworks are shown to be inconsistent or incomplete, I think the answer is clear: they would all become ridiculous. If we subscribe to Kantian ethics, we'd never be able to use such strong AI for any useful purpose, as it would be impermissible to treat such software as a means to an end, and not an end in itself. If we subscribe to Utilitarian ethics, we'd be stuck, as we don't have any way to know what pleasure or pain would mean for a non-sentient being. Etc., etc.

  • -1 That seems like rather a straw man of Turing's proposal to me. The whole point of TTT is that if a piece of software is capable of being indistinguishable from a human in conversation, who are we to judge it as insentient? The paucity of other potential criteria for moral personhood (Roger Penrose soul particles have yet to be found!) means that we would be arrogant to assume anything else. All this aside, my question was phrased in the conditional, and your entire answer is simply a flat denial of the condition. Mar 17, 2012 at 14:16
  • 1
    I don't think it is a straw man of Turing's proposal, rather an accurate representation of it. Turing's bar seemed impossibly high at the time, but now we are seeing that although it may be soon reached, we are still far from anything resembling consciousness. Furthermore, my answer pointed out that even if we did reach something resembling consciousness, we're still far from sentience. So, I'm not only denying the condition, I'm denying the conclusion even if we stipulate the condition. Mar 18, 2012 at 11:16
  • 1
    @Seamus: I thought I was doing that, via a reductio. Most ethical systems place a far greater burden on sentience than on mere consciousness for moral personhood; if we stipulate that AI capable of passing the Turing Test is sufficient for moral personhood, then shutting off such a machine would constitute murder. That seems, to me, to be an absurd conclusion, which would indicate that the stipulation is false. Mar 19, 2012 at 15:02
  • 2
    Arguably rebooting your phone isn't any more immoral than putting a person to sleep. Factory-resetting your phone might be a real moral bad if we had Turing-test-passing phones. But I still think this is beside the point. IF you thought passing the Turing test was sufficient for moral personhood, then you wouldn't reformat your computer. Any moral system that says "destroying moral persons is bad" has this consequence. I guess the question is which moral systems cope better with "low" bars to personhood.
    – Seamus
    Mar 19, 2012 at 15:42
  • 1
    In passing some of this reminds me of Egan's Crystal Nights short story -- the protagonist ends up making his simulated population "immortal" in order to assuage his guilt. You can read the story online here: ttapress.com/553/crystal-nights-by-greg-egan
    – Joseph Weissman
    Mar 19, 2012 at 16:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .