What is it referred to as?
You may be thinking about what's called a False Dilemma or False Dichotomy. The definition given in the lead on the Wikipedia page is:
A false dilemma is a type of informal fallacy that involves a situation in which only limited alternatives are considered, when in fact there is at least one additional option.
In your example, the false dilemma is that the student must choose between
putting the paper in the trash and
littering, when in fact there are more options, one of which is
However, when you add the fact that the professor knew the third option, then it seems to me to be less of a "fallacy" and more of a plain "trick" by which the professor causes the student to introduce the fallacy to themselves. If the student limited their consideration to only 2 choices, they forced the false dichotomy upon themselves.
Did the student in the example choose the best answer, as he only had two options?
The professor doesn't actually give any options, but only seems to imply some by the setup sentences. So, I would say that it is the student that actually introduces the fallacy, not the professor, and that the student picked from options that they presented to themselves.
Would you say that it is ok to reach out of the theoretical scenario given?
Certainly. Philosophers do this all the time, not just to respond to a False Dilemma. If someone says
If A then B. If B then C. but you can show a scenario in which A exists without B or without C, then you are demonstrating that their argument is not sound.
Is it fair?
Is this generally acceptable in philosophical debates? Are setup questions something that is common, or do some professors just use it because they can get away with it?
I don't know how to apply "fairness" to this situation. Sure, professors can "get away with it," but what if it's part of their teaching? Showing students that they need to fully evaluate a situation can be valuable for their education. It can also teach the student that, although they will learn how to identify the fallacy in this scenario, not everyone will reason correctly (just as they did not at first).
As for "philosophical debate," this may not apply. In philosophy, "debating" isn't exactly like arguing with someone in front of a crowd, such that tricking the other person might earn you "points" or help you "win." Instead, debating in philosophy has more to do with finding flaws in another view and presenting sound arguments in favor of your own. You could point out a False Dilemma fallacy in someone else's argument, or you could use questions rhetorically to highlight some weakness in their position, but you probably wouldn't try to trick them into answering a question incorrectly. Instead, generally, you are trying to persuade them by using sound arguments and identifying unsound arguments.