I was quite taken with David Deutsch's dismissal of the fear of the AI general intelligence singularity, based on the idea that any true intelligences will in some sense start with hybrid human-machine intelligences. I pictured human-machine cyborgs, or simulated human brains like the Human Brain Project is attempting.
But I just encountered Eliezer Yudkowsky's strong dismissal of this view:
I don't think that humans and machines "merging" is a likely source for the first superhuman intelligences. It took a century after the first cars before we could even begin to put a robotic exoskeleton on a horse, and a real car would still be faster than that.
I don't expect the first strong AIs to be based on algorithms discovered by way of neuroscience any more than the first airplanes looked like birds.
I don't think that nano-info-bio "convergence" is probable, inevitable, well-defined, or desirable.
I think extrapolating a Moore's Law graph of technological progress past the point where you say it predicts smarter-than-human AI is just plain weird. Smarter-than-human AI breaks your graphs.
The only key technological threshold I care about is the one where AI, which is to say AI software, becomes capable of strong self-improvement. We have no graph of progress toward this threshold and no idea where it lies (except that it should not be high above the human level because humans can do computer science), so it can't be timed by a graph, nor known to be near, nor known to be far. (Ignorance implies a wide credibility interval, not being certain that something is far away.)
I think outcomes are not good by default - I think outcomes can be made good, but this will require hard work that key actors may not have immediate incentives to do. Telling people that we're on a default trajectory to great and wonderful times is false. From
This throws the spotlight for me, on what would is involved in transmission of human intelligence, between individuals and to any AI. It has to be transmission in some sense. For instance with https://en.m.wikipedia.org/wiki/AlphaZero even though it was able to adapt from Go to Chess, this involves some intelligence about game design and aims. A container for calculations, in the same way that a human baby's brain and it's engagement with energy flows is a container, which must develop considerably to redefine the game.
Is there anything beyond speculation, about how much a true general AI intelligence would need language, training, and cultural transmission, or whether it could somehow become 'self-arising' - could it in some sense become an entire new evolutionary tree, developing itself as evolution developed us?
Perhaps there is a parallel issue about how much descendents resemble antecedents, how intelligence is preserved and transmitted, and how much we can be self-defining. There seems a tension between individual and community intelligence..