Do we only believe statements like "2+2=4" when we think of them? That is what I believe(at least at this moment!). I am not saying we believe it is false, either, but most of the time we are just not thinking of it, hence not believing it. I would be open to any counterarguments.
Yep. Just like running, speaking, throwing, climbing, ambling etc., beliefs are not something you have (vis. container/storage metaphors) it’s something you do. The same presumably goes for all cognitive attitudes: knowing, conjecturing, considering, remembering, inferring, suspecting, alluding, etc. See Eric Shwitzgebel for dispositional accounts of belief: http://www.faculty.ucr.edu/~eschwitz/SchwitzAbs/AccountBel.htm Also a good paper by Daniel Casasanto (at the top of the list) “All Concepts are Ad Hoc,…” http://www.casasanto.com/
As in so many matters, a clarifying answer to precisely this question is given by Wittgenstein. Belief, like so many psychological attitudes, does not have genuine duration. It is more akin to an ability or a disposition than a state.
For example, you car has 300 Horse Power. It has this regardless of whether it is moving or not. Its power is not a state but an ability. Understanding, belief and knowledge are similar.
I want to talk about a "state of consciousness", and to use this expression to refer to the seeing of a certain picture, the hearing of a tone, a sensation of pain or of taste, etc. I want to say that believing, understanding, knowing, intending, and others, are not states of consciousness. If for the moment I call these latter "dispositions", then an important difference between dispositions and states of consciousness consists in the fact that a disposition is not interrupted by a break in consciousness or a shift in attention. (And that is of course not a causal remark.) Really one hardly ever says that one has believed or understood something "uninterruptedly" since yesterday. An interruption of belief would be a period of unbelief, not, e.g. the withdrawal of attention from what one believes, or, e.g. sleep.
Remarks of the Philosophy of Psychology, vol II, sec. 45.