In particular, what ethical considerations might be involved in the predicted development of simulated minds possessing great verisimilitude to human minds -- or superhuman-level artificial intelligence?
I imagine that such a development will split the world of ethics in two, each side with vastly different ideas:
On one side, people will claim that such simulated minds, by virtue of their verisimilitude to human minds, have civil rights. Physicalism, especially the more complete kind (everything is physical, i.e. a Materialist view) is likely to attract many to this view. Physicalists will see the simulated minds as equivalent to human minds; since they view all phenomena to be physical, they will think of a replication of the human mind with advanced AI to be equivalent to the human mind itself, possessing the same neurological connections and storage in the form of transistors and memory.
On the other side, people will claim that despite the verisimilitude of these simulated minds to human minds, they do not and should not have any ethical rights. This is because they are just physical objects, and lack the consciousness and sentience of human beings. Dualists in particular will be attracted to this view, because they do not view the human consciousness as replicable by purely physical means. Therefore, regardless of how powerful the AI of computers are, they simply will not have the minds of people, and so one cannot attribute any particular rights to them.
Ultimately, it seems to me that the ethics of computers and advanced AI is highly dependent on one's ontological views; those inclined toward metaphysics would be more likely to deny rights to computers, while the pure physicalists would be more likely to attribute rights to computers.