On this very subject I found a 19-page paper published by the Machine Intelligence Research Institute (MIRI). Artificial General Intelligence and the Human Mental Model, Roman V. Yampolskiy & Joshua Fox. I suggest a keyword search on the title to find a link to the paper.
If we define intelligence on the human model, then intelligences will tautologically have many human properties. We instead use definitions in which intelligence is synonymous with optimization power, “an agent’s ability to achieve goals in a wide range of environments” (Legg 2008).
Still, if the purpose is to consider the effects on us of future superintelligences or other
non-human intelligences, definitions which better capture the relevant features should
be used (Chalmers 2010). In the words of Dijkstra (1984), the question of whether
a machine can think is “about as relevant as the question of whether Submarines Can
Swim”—the properties that count are not internal details, but rather those which have
effects that matter to us.
Nonetheless, the intuitions for the concepts of “mind” and “intelligence” are bound up with many human properties, while our focus is simply on agents that can impact our human future. For our purposes, then, the terms “mind” and “intelligence” may simply be read “optimizing agent” and “optimization power.”
The basic idea is that a superintelligence can emerge from human efforts to create GAI and human minds might not be able to comprehend such intelligence. This incorporates the idea that an agent acts with a goal to optimize a payoff or reward; and the agent also acts to increase its power of action in the service of attaining its goals. But these general ideas about intelligence are still limited to patterns originating as human intelligence.
Humans might infer that a superintelligence or alien intelligence has goals and "wants" the power to act so it can optimize some reward but that does not mean we will comprehend the goal or powers of action of every possible general intelligence.
The Three Laws of Robotics (Published in I, Robot by Isaac Asimov) would be human imposed rules that constrain the goals and power of action of otherwise autonomous agents (robots) that have the potential to benefit or harm humans. The explicit programming of robots to follow the rules seems to be impossible because lawsuits pop up constantly for humans complaining of harm caused by other humans. Humans giving orders to robots who can follow those orders would be interfering with the will of other humans causing something like robot rage instead of road rage.
Alan Turing asked whether a "machine" can "think" and wrote a paper including some methods called the Turing Test. This is an 86-page bullet-type presentation of related concepts:
The Turing Test is basically a human asking questions, without being able to see how who or what generates the answers, and deciding by the answers whether the response is generated by a human or machine or process that can think.
Three Laws of Robotics:
A robot may not injure a human being, or through inaction, allow humans to come to harm.
A robot must obey orders given to it by humans except where such orders would conflict with the First Law.
A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.
I would infer that a robot (machine intelligence) that could follow these rules would have human-like ability to think. But I would not recognize the machine as a human-like subject. To be a human-like subject the machine must interact with humans as if it were a fellow human. I would infer that a human or human-like subject comprehends the meaning of the rules in the human context and would be capable of following the rules or of violating the rules just like a human.
See the movie: I, Robot (2004). There are dumb robots that seem to be able to obey the rules, however, the programmers or engineers must anticipate what is or is not harmful to a human being in every possible circumstance, and what is or is not the meaning of "obey orders given to it by humans", and it seems to be logically impossible for programmers or engineers to code this into a machine that operates with human-like general intelligence. Sonny is a humanoid machine who interacts with humans more like an intersubjective human being and he is obviously not programmed or engineered to follow The Three Laws. VIKI is the central intelligence who learns to interpret The Three Laws. VIKI decides to protect humans by imposing restrictions on their liberty. This expresses the human pattern of drama called paternalism.
In terms of human rules, the law imposes a duty to avoid causing harm to others. This duty relies on our intersubjective experience of what is harmful or not harmful to self and others. The law typically does not impose a duty to rescue others or to prevent harm to others via inaction. In the United States, a person is either obligated to provide aid via an employment contract or is considered to be a Good Samaritan who voluntarily comes to the aid of others because there is no legal duty. A robot that must go to the aid of others due to its programming might be very human-like but would not have the rights and duties of a person.
As a general rule, one legal person (human) does not have any legal duty to follow the orders of another person. Exceptions apply when one contracts for employment within the scope of duties; when one holds public office within the scope of duties; and when one serves in the military within the scope of duties even including a duty to disobey an unlawful order. Legal remedies for failure to obey or disobey orders vary by the context in the records of legal disputes.
General machine intelligence must become almost human such that it evokes our sense of intersubjectivity. Then we would begin to think that the machine thinks and acts like a human being. In the imaginary dramatic context where we experience a genetically enhanced animal or artificial intelligence as a fellow subject, and not something similar to yet distinct from a human subject, I have no idea how to separate the concept of anthropomorphism from GAI.