The tricky part of an argument like this is the proof of Utility(X). This argument works very well in situations where the utility is obvious (I either do God's will by helping this orphan child, or I make myself feel better by helping this orphan child), but it gets more difficult to assign meaningful utility values in other situations. We humans are not very good at estimating these sorts of values (is 54627 * 183059 - 2,984,120 * 1 greater or less than 0?) This makes the wager tricky.
However, even if we are given our Utility function, its still tricky, because we have to use that function after it has been proven in order to determine if any action is good or bad. Thus our Utility function must include the utility costs of applying the formula. Otherwise we may spend forever in a chaotic loop around U(X) = 0, unsure how to proceed. The Utility function must be self-referential to avoid this issue.
The fact that you are going to use probabilities on these utility functions implies you need the basics of arithmetic. This doesn't sound like anything too important, but it's not a trivial detail, as we soon find out.
Now the final question before things get crazy is: do you have to be able to prove the validity of your results? If you are willing to take this on faith, as most if not all religious beliefs are taken, you can rest assured that this sort of logic can be made to work. However, for the faithless, who must rely on hard mathematics and provability, the concept gets squirely fast.
Why? Well, if you put all of these pieces together, you have the basis of Tarski's undefinability theorem, and whenever his name gets involved, you know you're in the really messy bits of logic and mathematics. His theorem basically states that no self-referential system which can prove all truths of mathematics (read: contains the Peano axioms of arithmetic) and contains negation can define its own semantics. In particular, it cannot define what "true" means within the system. His proof involves the diagonal lemma, which makes all sorts of things frustrating, but it ends up using the system's descriptions of arithmetic to form inconsistent results. The way this occurs is specific to the system, but Tarski proved that it must occur in every system satisfying the ruleset he was exploring (and its surprisingly easy to accidentally support that ruleset).
This means that you get stuck with the interpretation of the Utility function as being handled outside of the language of said functions (a metalanguage feature). This creates all sorts of interesting loopholes that are impossible to plug within the Utility function.
There are ways around this. I've enumerated a few in my own exploration, but the key is that you have to describe the Utility functions in a language which sidesteps Tarski's work. If you accidentally step into the languages he described in his undefinability theorem, that step seals the fate of not only the resulting Utility function, but the entire class of Utility functions created in that form.
- You can design the system to refuse to admit the diagonalization lemma. Dan Willard has created a class of curious self-verifying theories which are intentionally too weak to support diagonalization (in particular, multiplication is not total in his theories). Wisps of this sort of thinking show up because Dan's systems start with an infinity (everything), and subtract and divide from there, rather than starting from 0 and adding/multiplying, and I've seen several belief systems which focus on being "part of everything."
- You don't have to prove your system. This can be undervalued, but remember that nobody has been able to demonstrate a proof of their religion which is satisfactory for all other people. Faith is kind of a big deal for religions.
- You can admit uncertainty. If your system is not complete, meaning it does not admit a proof for everything, you can avoid the provability issues Tarski addressed. The phrase "if it is God's will" comes to mind.
- You can omit negation from logic. Negation is a key part of Tarski's proof. If you never negate anything, his work cannot disprove yours.
- You can weaken the wager slightly. If you can demonstrate that U(X) >= 0, whether or not there is a God (i.e. it's not bad, no matter what), then it is trivial to show that U(X) cannot be less than 0. "Do what you will, may it harm none"
I'm certain there are others, but those are the ones I have found interesting.