Is there any reason to think that GAI, in every conceivable form and with any possible safeguard, will annihilate any civilization that builds it, thereby solving the Fermi paradox (assuming that every civilization must build it for space travel)?

I can't see why, unless GAI is sentient, and sentience intrinsically, with enough computing power, annihilates everything as soon as it emerges. While this may solve the Fermi paradox, I personally believe GAI won't be sentient anyway, though I guess we'll see (or not).

I don't like the idea that flawed but powerful processing power alone is enough to kill everything, as everything contains bugs. (Life contains unhelpful mutations but still exists, is increasingly complex, etc. My physics is also bad, but if all systems contain random noise, then complexity alone should not suffice?)

I'm interested not in whether this solves the Fermi paradox (which would be scary) but if the Fermi paradox can seriously add anything to 'the dangers of AI'.

Whether or not the fermi paradox has any relevance to the dangers of GAI should depend on how likely any civilization (including ours) will be able to contact another after developing GAI.

The Drake Equation is part of why there are official searches for alien life... 12,500 intelligent alien civilizations which may currently exist. This is just a guess [for the milky way].


Supposing p=1 (which it isn't), then the Fermi paradox would imply that the chances of annihilation from GAI are large1, though I am not proficient enough at statistics to estimate it.

  1. We can be 95% sure it is over 1/3000.
  • 1
    This has been discussed at length with Bailey in Could AI be the Great Filter? (this made much noise in the media), Naude in Extraterrestrial Artificial Intelligence arguing for AI as the great filter, Less Wrong arguing against it, etc. But the rampant anthropomorphism involved in assessing alien civilizations makes all arguments of this sort not very credible.
    – Conifold
    Jan 25 at 12:57
  • There are many dangers with AI, or perhaps less alarmingly: there are many important ethical questions with AI. But these important ethical questions are all about how humans are using AI, not about the risk that AI might become conscious and become a dark overlord. However, some people dislike it when we discuss ethical questions related to AI, so they like to divert our attention by derailing every serious discussion of AI ethics into a discussion of possible AI overlord apocalypse.
    – Stef
    Jan 25 at 23:49
  • You might be interested about Vernor Vinge's ramblings about a coming "AI singularity"
    – Stef
    Jan 26 at 0:17
  • GAI destroying the civilization alone won't solve the Fermi paradox, because then where is the GAI that has destroyed the civilization? IMO, you'd need to dig deeper, e.g. it takes the form of grey goo, but which consists of such small bots dispersed in space at sufficiently low density that our telescopes can't notice it.
    – rus9384
    Jan 28 at 14:13
  • yeah, maybe that does make sense @rus9384
    – user71226
    Jan 28 at 20:57

4 Answers 4


I suggest the terrifying danger you foresee might be avoided by taking the precaution of wiring an 'OFF' switch into the evil bot's power supply.

  • 4
    I'm a novice, Marco, but is an off switch possible if AI is decentralised? And would it be introduced prior to its ability to recognise and disable it? 'It' makes it sound like a solitary thing, whereas it seems as though it might be a more pervasive entity. No idea what I'm taling about though. Jan 25 at 12:05
  • 3
    @Futilitarian you're right, the feasibility of an 'off switch' depends on how the ai is implemented, and truly no matter how it's implemented, there's probably a path for any true superintelligent general AI to get control of its own off switch eventually.
    – TKoL
    Jan 25 at 12:11
  • @MarcoOcram I do not understand why the Fermi paradoxon, GAI and a possible interaction of both is a subject for philosophy – don’t you think it would be a good subject for your next novel? Or is this novel already in print :-)
    – Jo Wehler
    Jan 25 at 13:02

We already see current expert systems "solving" problems by exploiting "glitches" in the simulation rather than finding the best solution in the scenario that we've prepared for them. So it's not really about the dangers that we foresee it's about the dangers that we don't foresee.

You could also have discussions about what if it turns conscious and develops concepts of pain and death and how to avoid them... And yes we might use those in regards to "reinforcement learning" though we don't expect them to be "real", but what do we know...

Also an OFF switch might be insufficient. Like hook that up to the internet and it might store it's "brain state" somewhere on a server in you-don't-know-where or encrypt it on it's own hardware, obfuscated in regular data and the moment you boot it up again, the same is back on or worse.

The thing is on the one hand we want these things to be all-powerful to be useful to use on the other hand we are afraid of what that would look like because if it ever would have gotten to the point we are likely not even able to think of all the ways it could avoid us trying to stop it.

Though why would you build a GAI in the first place other for a proof of concept?


Less wrong says

"this disaster can’t be an unfriendly super-AI, because that should be visible"... an UFAI should also be counted as an 'expanding lasting life'

Why would UFAI be any more visible than an alien civilization? Given that ET's have AI it seems unlikely that the visibility of UFAI is greater than ET's.

What shifting the paradox from ET's to UFAI's makes for could be called the "less alien paradox": why haven't we seen UFAIs? To say it is more problematic than the fermi paradox seem wrong, especially supposing that ET's, unlike UFAI's, are not necessarily unfriendly. So it may help resolve the fermi paradox.

Indeed, if we annihilate ourselves then it seems fair to assume - at this point - that AI has contributed to the filter (socially or UFAI) for other civilizations to find us. Whereas to claim that our civilization will not be annihilated suggests that there are ETs.

This is just a fun way of saying that existential risks definitely seem relevant to whether or not we will make contact with (and it seems fair to think that you can contact - in some sense anyway - anything we would call "life" if not "intelligence") ET's and resolve the paradox etc..


I'm not convinced this is a useful line of inquiry. Neither the Fermi Paradox nor the risk of a hyperintelligent unaligned AI are universally accepted as real problems in their respective fields:

  • For the Fermi Paradox, it's fairly obvious that most of the terms in the Drake equation are completely unknown, and probably unknowable. There are many regions of the Drake configuration space in which we should not expect to observe aliens, and which are also compatible with our existing models of cosmology. Based on current observation, it would seem that we are in one of those regions - but that's not a "paradox," it's just an observation.
  • For AI, there are many different problems. Here are a few:
    • There is no mathematically precise definition of intelligence which extends far beyond the capabilities of humans. Intelligence is normally defined based on a clustering analysis of intelligence test results. This analysis shows that many different tests will tend to correlate with one another, suggesting that there is something to be measured (and the tests are not just axiomatically "valid"), but this is an operational definition. It becomes meaningless when we start to talk about anything significantly smarter (or dumber) than the humans who originally took the test. This is not to say that hyperintelligence is definitionally impossible, merely that it cannot be formalized and precisely reasoned about given the currently available research. But that's already a serious problem for a significant cohort of the AI safety movement, because they want to mathematically prove that a hypothetical "friendly" AI is not a danger to us, and a formalization of hyperintelligence would be, at the very least, rather useful for that purpose.
      • This idea also runs into various metaethical problems, but let's not get too far into the weeds here.
    • Some AI safety folks seem to believe that intelligence (whatever it is) is positively correlated with the ability to build more intelligent AIs, leading to a positive feedback loop and runaway progress once some threshold is crossed. The problem with this is that, since intelligence is not formalized, this is at best an intuition, and at worst an outright guess. Very smart people have been trying to build rule-based AIs for a very long time, with almost no success whatsoever. Modern machine learning AIs do require some intelligence to design their basic architecture, but much more important is raw computation and data. It is possible that there is some algorithmic upper bound on how "good" a machine learning architecture can get before it has saturated the available data and computation. There is, after all, a finite amount of data in any given dataset, and trying to extract more than that will lead to overfitting. If this is the case, then smarter AIs will not beget smarter AIs, and the positive feedback loop fails to exist (or runs out at some point).
    • In the more immediate term, there is significant disagreement over whether we should focus more on making existing AIs better align with human values, or if we should focus on the formal proofs I mentioned earlier. These divergent preferences arise from very different ideas of what "AI risk" entails, how likely various different outcomes are, etc.

In short, you are asking whether one questionable premise may be used to explain or justify another questionable premise. I would say that we lack sufficient understanding of (and evidence for) either premise to seriously evaluate this kind of relationship.

You must log in to answer this question.