Alice and Bob are playing rock-paper-scissors. Alice typically throws rock. Bob, while thinking to himself, is naïve and wants to throw paper, but also knows that Alice is sneaky and so will anticipate that Alice will throw paper and chooses to throw scissors. So Bob throws paper. Alice, however, is double-sneaky and knew that Bob would anticipate Alice's sneakiness and throws scissors.
What kind of reasoning is this?
I've heard it referred to as 'wine in front of me' reasoning, referencing the scene in The Princess Bride where a villain places two chalices filled with wine in front of the protagonist. One is poisoned and one is not, and the protagonist must decide whether to switch his cup with the villain's while the villain is not looking.
It is generally considered to be a pointless line of reasoning, because it reaches no conclusion and the depth at which it stops can be considered effectively random for any pair of people.
My first thought was deductive reasoning, with the subgroup of reverse psychology as the specific type. But I think predictive reasoning might be a better way of putting it. Besides, reverse psychology is the principle or practice of subtly encouraging a behavior or belief by advocating its opposite, which is not what is happening in your example.
You are trying to predict the opponent's action, and act out from that. While there may be layers, the fact is that you are reasoning your way to what the opponent will do by predicting it. Reverse psychology would have a play in it if Alice was talking about how clever she was and chose to then do scissors in anticipation of her sneakiness being accounted for.
So deductive and/or predictive reasoning, I would say.
I thought it was called the "Sicilian Game of Wits" and was typically not played with "Rock, Paper, Scissor" but, instead, with two glasses of wine, one lace with iocaine powder.
As @Bread suggests in a comment mathematical models of this type of behavior would be studied in Game Theory. Citing Roger B. Myerson's Game Theory: Analysis of Conflict, Wikipedia describes Game Theory as
Game theory is the study of mathematical models of strategic interaction between rational decision-makers.
This provides a clue for what one might call the kind of reasoning exhibited by Alice and Bob in the rock-paper-scissors example provided by the OP.
One could think of the kind of reasoning they are engaged in as strategic interaction between rational decision-makers.
Wikipedia contributors. "Game theory." Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia, 20 Mar. 2019. Web. 21 Mar. 2019.
