What is the name of this kind of reasoning? (Not a logical fallacy.)

Alice and Bob are playing rock-paper-scissors. Alice typically throws rock. Bob, while thinking to himself, is naïve and wants to throw paper, but also knows that Alice is sneaky and so will anticipate that Alice will throw paper and chooses to throw scissors. So Bob throws paper. Alice, however, is double-sneaky and knew that Bob would anticipate Alice's sneakiness and throws scissors.

What kind of reasoning is this?

• It's a type of game theory, but I don't know the specific name for it. – Bread Feb 17 '19 at 19:49
• Its called 'reverse psychology' that it's done twice makes it 'double reverse psychology', or reverse reverse... or 'ironic' reverse psychology. – Richard Feb 17 '19 at 19:50
• I actually played a similar game with my brother and nephews , the game was like this : you decide to hide a coin in one hand, and I have to guess in which hand you hid it, but your choice should not be random, you actually have to think to figure out what hand I will choose next time, and the same thing, I try to predict which hand will you hide it next time... What I discovered is that when the game is random, I get it about 50% of the time, but when the game is psychological and involves thinking and strategy, then I get it about 80% of the time, sometimes 6 to 7 times in a row. – SmootQ Feb 17 '19 at 21:47
• Wikipedia calls it backward induction. It is a typical way of reasoning in game theory. However, in circular games, like rock-paper-scissors, it leads nowhere. If both players are "infinitely sneaky" the "best moves" oscillate depending on the depth of induction, one might as well play randomly. – Conifold Feb 18 '19 at 5:22
• As a matter of interest, back in the 1980s some AI researchers wrote a program to play rock-paper-scissors. It analysed its opponent's choices and looked for patterns. Humans are quite poor at generating random sequences, and after a training period of about 100 games against any single opponent, the program was able to achieve a win rate of about 50%, whereas with random play you would expect to win/draw/lose a third of the time. – Bumble Feb 19 '19 at 8:35

I've heard it referred to as 'wine in front of me' reasoning, referencing the scene in The Princess Bride where a villain places two chalices filled with wine in front of the protagonist. One is poisoned and one is not, and the protagonist must decide whether to switch his cup with the villain's while the villain is not looking.

It is generally considered to be a pointless line of reasoning, because it reaches no conclusion and the depth at which it stops can be considered effectively random for any pair of people.

My first thought was deductive reasoning, with the subgroup of reverse psychology as the specific type. But I think predictive reasoning might be a better way of putting it. Besides, reverse psychology is the principle or practice of subtly encouraging a behavior or belief by advocating its opposite, which is not what is happening in your example.

You are trying to predict the opponent's action, and act out from that. While there may be layers, the fact is that you are reasoning your way to what the opponent will do by predicting it. Reverse psychology would have a play in it if Alice was talking about how clever she was and chose to then do scissors in anticipation of her sneakiness being accounted for.

So deductive and/or predictive reasoning, I would say.

I thought it was called the "Sicilian Game of Wits" and was typically not played with "Rock, Paper, Scissor" but, instead, with two glasses of wine, one lace with iocaine powder.

As @Bread suggests in a comment mathematical models of this type of behavior would be studied in Game Theory. Citing Roger B. Myerson's Game Theory: Analysis of Conflict, Wikipedia describes Game Theory as

Game theory is the study of mathematical models of strategic interaction between rational decision-makers.

This provides a clue for what one might call the kind of reasoning exhibited by Alice and Bob in the rock-paper-scissors example provided by the OP.

One could think of the kind of reasoning they are engaged in as strategic interaction between rational decision-makers.

Wikipedia contributors. "Game theory." Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia, 20 Mar. 2019. Web. 21 Mar. 2019.