This answer is within the context of a Bayesian worldview, which, as has been explained in other answers, basically associates a probability with any and all statements you can come up with.
What would it mean to talk about the probability of God?
A very simple way to think like a Bayesian is to ask yourself: how much would you bet on the proposition being true, to expect a positive outcome? In this particular case, there is the famous Pascal's Wager, which says that it is safer to bet on the existence of God because you have to invest relatively little (a little bit of time and maybe some money for donations for your church...), but can gain an enormous amount (eternity in heaven). Or if you don't believe in god, you are betting the little time and money you safe in life vs. an eternity in hell.
Can this be calculated?
No. By design, everything and anything that is believed about God is such that it is not visible, falsifiable, or in any other way open to objective criteria. Almost by definition, anything that is up for objective research, logic, or anything of the kind (i.e. anything that could go into a calculation) is not part of the volume of believes about God.
As with most statements, it is impossible to assign a probability of 0 or 1, because this would mean we knew things that we cannot (by design) know or not know. This is not a special property of this proposition, but goes for almost anything a Bayesian is thinking about.
On the one hand, it seems intuitively ridiculous to assign a mathematical figure to the existence of an entity. On the other, even if we can’t attach a figure, can we not at least discuss its relative probability?
Sure, every Bayesian is free to come up with a subjective probability they subscribe to, for any given statement. It is not so much about the "objective" probability. It is "normal" that different Bayesians assign different probabilities to the same question, simply because it usually depends on what knowledge the person has available at any point in time. For example, for me it is relatively simple to assign a probability to "will it rain in 30 minutes in the city I am in right now" by looking out of the window and witnessing perfectly blue sky, 30°C and zero clouds, in an area that is not usually experiencing sudden weather changes. But I could not say anything with confidence about the weather in some city on the other side of the globe, unknown to me, because I simply do not know the local weather patterns. If I have access to a smartphone weather app, my surety would increase. And so on and forth.
Heck, for the question of whether God exists, there even is a formal (albeit possibly slightly tongue-in-cheek?) scale, in the form of the Spectrum of Theistic Probability.
For example, no one knows the probability of me becoming the most famous musician ever.
Oh, but we do, at least on a broad spectrum. If we know nothing whatsoever about you, then we can simply divide "you" (i.e., 1) by the number of humans, and get in the rough ballpark of a probability of 1/8.000.000.000. Knowing that you have a computer or smartphone probably catapults you into the 1/1.000.000.000 area (just to pull numbers out of thin air). Instead, if we knew that you are at least in the top 100 musicians right now by some metric, say your place on Spotify playlists, we could relatively conservatively just assume 1/100 as a chance. The more we knew about you, the better the result would be (especially if we knew anything about your musical career so far). We would never, in any case, assign a probability of 0 or 1.
But clearly, it seems smaller than me rolling a six on a die.
Clearly, yes.
Can one do this with God?
Sure!
For example, I consider myself at roughly 6.99 on the Spectrum mentioned above (and the only reason it's not 7 is because 0 or 1 are not valid probabilities in my particular worldview, for almost any question). In quite a few decades of life, I have not witnessed anything whatsoever that speaks to me as counting towards the truth of that statement. Conversely, for all "proofs" or "signs" that religious people have come up, I can easily think of purely logical, objective, psychological etc. explanations. I can easily think of ways how a religion can manifest some thousand years ago and grow to what it is today, with no mystical influences whatsoever. I have access to logical arguments along the line of "there are at least X major religions, everyone of them using the same arguments to tell me that they are the only true one, so..."
If, tomorrow, I get abducted by a literal arch angel, be transported into Heaven, talk with God and the Devil, get super powers of wisdom and capabilities and brought back as the next Saviour, then my personal believes could easily change (I assume). I mean, I would still need to figure out for myself whether it's just aliens, whether I live in a simulation instead, and so on and forth, but my position on the Spectrum would probably shift just a teeny tiny little bit, at least.