Consider the following scenarios, each of which at their core refer to a form of 'reward hacking'.

Based on personal observation, each scenario below generally elicits an immediate, visceral feeling of ~'this is just wrong' in the majority of people.

The question is: Which ethical framework would deem 'brain reward hacking' immoral? And why?

The scenarios:

Access to medical monitoring equipment and an unlimited amount of pure morphine

One can imagine a device that monitors a person's blood morphine levels, measures brain activity and vitals, and continuously injects the precise amount of morphine that keeps the attached human in a constant state of 'happiness' (without killing them).

Soma in Brave New World

The book describes a fictional society where a 'happiness drug' called soma is provided to citizens, with the aim of keeping them 'happy' and free of 'negative' emotions. When people become unhappy about something, the government simply administers more soma.

Super-intelligent AI that is a human happiness maximizer

Suppose a super-intelligent AI was developed with the goal of maximizing human happiness. Upon learning a bit of neuroscience and engineering, the AI decides to and surreptitiously implants all humans with electrodes and sensors. Each implant monitors each person's 'happiness' levels and directly stimulates the brain reward centers to keep the humans in a perpetual state of maximum 'happiness'.

Ethical Analysis

Utilitarian: It seems that all three scenarios would be a net-positive increase in happiness, thus not immoral.

Rawlsian: If any of the three scenarios were to become more likely, I am not sure that I would be particularly averse to being born as one of the people described in the scenarios. If I was born as one of them, it seems that I would have a relatively blissful, happy existence.

Kantian: If these were universalized, and everyone was on morphine or soma all the time, I can see that it couldn't go on for too long -- someone would need to operate the morphine/soma factories, get raw materials, machines, etc... However, in the case of AI, the AI would perform those tasks, and all humans could theoretically stay in a perpetual state of 'happiness'. So under Kantian ethics, unlimited soma and morphine scenarios would lead to a contradiction, while the AI scenario would not (?).

Concluding thoughts

It seems that the disgust/'this is appalling' factor primarily arises when the scenarios are considered from a third-person perspective. The person who has the morphine/soma/electrode treatment done to them experiences happiness, while a person who hears or sees this being done to another person feels disgusted.

So then is the moral intuition about brain reward hacking irrational?

  • 2
    The same person is perfectly capable of assessing the scenarios "from a third person perspective" and getting "disgusted" (until they are on morphine). The whole analysis is based on infantile ideas about happiness and how things work, from indulging in instant gratification to big brother AI taking care of all problems. Even utilitarians like Mill had enough sense to distinguish "higher pleasures", and mental degeneration under the care of machines was described as far back as H.G. Wells. – Conifold Aug 15 at 4:41
  • 1
    Nice q! Upvoted. I've no answers though. Other than to say/suggest that the reason for disgust is we're already generally onto the "morphine" . It's called Hollywood (metaphorically). When you (males more likely) come out of a James Bond or Rambo show you are – consciously or subconsciously – fantasising that you are that hero. – Rusi-packing-up Aug 15 at 7:21
  • 1
    On Mill see SEP Happiness and Higher Pleasures and Bhardwaj's Higher and Lower Pleasures and our Moral Psychology. Wells's describes the degeneration of the Eloi in The Time Machine. – Conifold Aug 15 at 23:05
  • 1
    Do a search for: " 'Artificial Utility' in Ethics," "the Wire-head Problem," "Ethical Utility Counterfeiting." We can work to get the best answers to this question as possible. – Tautological Revelations Aug 17 at 20:36
  • 1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.