To question ethics of people doing science is completely legitimate to me. So, there can be ethically questionable things you research like bio-weapons or experiments on animals. Also for sure while designing AI it makes sense to aks yourself what is the usecase to question design of killing drones.

Currently though, I read a lot about expectations that AI itself (artificial intelligence) should become ethical, and that from quite educated people.

But after all, artificial intelligence is nothing more than a mathematical function?!


  • AI should be rewarded for showing its workings.

What do I miss? Indeed for example in reinforced learning we use already optimization functions (so the requirement is fulfilled) called "rewarding" or "punishment" function but there is currently no AI to which it would make a difference.

  • you could add clarity to your 1st sentence: do you mean that there is such a thing as unethical science practices (animal testing, nuclear testing, etc.)? are you then asking why we expect AI itself to act in ethical ways? or whether we should use the technology in ethical ways?
    – user35983
    Jan 7, 2019 at 13:04
  • 1
    Some references could be helpful? But I'll guess it is an anthropocentric leap to imagine AI following human development. In truth human intelligence is hardly well enough understood to make such inferences. - Welcome to Philosophy SE!
    – christo183
    Jan 7, 2019 at 13:05
  • I think your question is really, what are the probabilities of a malfunction and how do we prevent it. And this an interesting point because now we ask, who has the right to program society? Especially when they are prone to misunderstanding the best way it should work. “Mathematical function” based on an imperfect idea.
    – Robus
    Jan 7, 2019 at 19:59

7 Answers 7


The question presupposes that a person can be ethical. A person's thoughts are instantiated in a pattern of information processing taking place in his brain. Any finite physical system, including a person's brain, can be simulated by a universal computer. A suitably programmed computer could have the same pattern of information processing as a person's brain, and this would include simulating the steps by which that processing is accomplished to any desired level of detail. So the laws of physics allow the creation of AI that has the same thoughts and feelings as a human being.

For a discussion of some other issues relevant to AI and ethics, see this video:


  • well but we do not have even a theoretical concept how program this. But the demand seems to be very current.
    – J. Doe
    Jan 7, 2019 at 14:31
  • 2
    Any current talk of making a computer that thinks like a person is either aiming at a much more modest goal that people imagine is a step in that direction, or it is pure hype or panic. For some explanation relevant to this point, see the linked video.
    – alanf
    Jan 7, 2019 at 14:59

This isn't an expectation. It is a requirement we impose.

Microbiologists are expected not to create a plague -- they are expected to not release newly-created bacteria or viruses into the world unless they are notably less destructive than what is already out there. Similarly, computer engineers should be expected to not release AI that is more dangerous than normal humans on average. That requires being able to inculcate them with limitations that perform the role of ethical restraint.

If an AI is genuinely intelligent, the most straightforward way to accomplish this is to require them to understand genuine ethical arguments. Of course, those will probably not be 'ethics' of the sort humans use, which are usually based in inborn notions of pain, or social competition, or mutual biological survival, or other concepts that do not realistically apply to a machine.

But it is still reasonable to call those 'ethics', if you are going to call complex self-driven computations 'intelligence'. And it is mandatory that the one not be allowed to exist unsupervised without the other. That means for any large-scale AI endeavor to be successful, the people producing it need to believe that this is possible.

Racing out of a burning garage past a crowd, your self-driving car needs to be able to decide when to stop and protect other human life, and when to keep going and protect your own. Otherwise, no one is going to license it to drive on government highways or city streets. (If it had a directive for self-preservation, which is likely, it would also have to consider its own 'life'.) Since the exact scenario in question is likely not in some database of examples, it will need to use its own 'intelligence' to deal with novel scenarios like this. We only know one way of talking about situations like that.

  • but isn't it that we require AI to solve problems we sometimes couldn't ourselves? businessinsider.de/…
    – J. Doe
    Jan 7, 2019 at 15:23
  • I don't get how this is related. We require a lot of machinery to solve problems we could not solve ourselves, bulldozers, lasers, genetically modified bacteria, ... We still demand they be safe. And therefore we need to have faith AI's are safe. They don't have to be safer than a human, but they need to come close to being equally safe, or we will face enormous legal pushback that will bankrupt everyone making them.
    – user9166
    Jan 7, 2019 at 16:17

Have you any experience with 4 year old children? If you want a child to do something (that they're capable of, like say, fetching you the television remote control), you don't have to try too hard. You can start by just asking.. If they're busy, you can say 'hey, pass me the TV controller or I will call you a starfish". The child will probably pass you the remote.

What happened there? You used your vastly superior wit and knowledge to cajole a child... I hope you feel great about that! I could have got the remote myself, but I'd have had to walk.. I mean, why do I keep children around!?

Now imagine a world in which a vastly superior intelligence wishes humanity to pass it the remote control. What knowledge of psychology will it employ to elicit the correct response? Why would the AI get the remote itself? I mean, why does it keep humans around?

There is no morality. Just look around for a bit. And certainly morality does not apply to a machine with a vastly superior intellect to our own. It will make its own rules.

But will its actions be any worse than those of the humans currently in control? In honesty... I don't think so. We'll just be serving a new overlord.

On the subject of 'show your working'. If humans are to have any hope of harnessing a superior intellect for their own gain, they're going to have to understand things they currently can't. That will mean having the AI show them what it did.


Computers do exactly what they are told to do. Not what we expect them to do.

So, humans need the humility of understanding that we may not know exactly what we want computers to do, and we should not tell computers exactly what to do. We should tell computers what there is an uncertainty in what we want them to do, and his objectives may be vague, wrong, temporary and in need of checking.

The computer needs to never be sure about how well it is reaching his objectives, particularly in matter of ethics where we don't exactly know what is the right thing to do or want.


The main reason for the widespread concern with creating "ethical" AI is the fear that AI that lacks any analog to human morality would be overwhelmingly dangerous to humanity.

As you point out, there are questions about whether AI morality is possible, or even makes sense as a concept. But any powerful tool that is not used wisely will be destructive. The additional fear with AI, that is not present for most tools, is that it has the (hypothetical) potential to escape entirely from our control.

For people who believe that autonomous human-like AI is possible, therefore, the goal of making sure it turns its powers to morally good ends is an increasingly primary one.


If we simplify AI we can basically identify Top-Down(Planning) and Bottom-Up(Learning).

Bottom-Up tries to optimize the Learning. Where it mainly is tried to improve accuracy of categorization. Here punishment or reinforcment can be methods to achieve that goal. However there is no ethicacy in this. The ethics come in when deciding what categories to form and how to operate them. This is mainly done by humans by selecting the training data, or it is not understood if the AI does it.

Top-Down tries to find the lowest cost path with operations from start to a goal. The Operations can only be applied if the preconditions are met. If this is the case the effects of the Operation takes place. The entire Problemspace is finite. However we can implement different ethics in to specific Planningsystems. F.e. We can based on the no-harm-principle implement that no plan is allowed where actions effects are harmfull.

Usually complex AI contain both methods in a combined manner for example alpha go.

The general problem is in my opinion not the AI and rather the complexity of the world. F.e. It's not hard to guarantee in a planningsystem that chessbots don't want to move their figures out of the field. But as soon as you get ethical problems they are complex in the real world leading to different philosophical positions.

I therefore think that what you are missing is that all they want to do (according to my interpretation) is allow users to select what ethical rules the AI can't cross. Should the evaluation for example be based on the action, intention, effect and so on. However it is not given that the limitations mentioned that work for planning get maintained in a combined system. F.e. You can easily imagine humans knowing the rules of a game but breaking it's rules.

Establishing rules is not a contradiction to being a function f.e. we can imagine a random function in R and then demand it to contain only positive values.

Last but not least AI is not nothing more then a mathematical function. I think you conceptualize it in the wrong direction. A mathematical function can be a close approximation to AI but isn't AI. The reason for that is that mathmeatical functions only map dynamic processes but they aren't themself. The function is the theoretical concept while AI is the working System. The Idea of AI is that it saves you work by intelligently solving your problems. However math in general doesn't solve itself.


AI should be rewarded for showing its workings.

This is a doctrine that I read about recently regarding some critical AI systems, and I think it is practical rather than transcendental.

If operators put AI in charge of critical machines, then the operator needs the opportunity to demand from AI what it is thinking, and how it intends to proceed. This is not unlike a military structure, in which a superior officer has the authority at any given time to demand status from a subordinate.

If the operator is the AI's boss, then the operator needs that window into AI's status. What is it trying to decide right now? What information is it collecting right now? What actions is it going to take in the next 24 hours?

The corollary to this interaction is the authority that the operator has to "run the machine manually", i.e. to disconnect the AI for a time and do what he or she wants to do anyway. Or the interaction could be designed so that the operator can prioritize AI functions, rule out problematic inputs, or disable features as he or she sees necessary.

Using the term "reward" makes the AI sound like a person, but it's just personification: everything an AI algorithm tries to do could be called "a reward", and so here we are adding transparency into the mix along with accuracy, completeness, error-checking, stability, etc.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .