No fallacy there. In fact, it's quite rational thinking. Communication is not free, it costs resources -- especially time. It is rational to pay attention to the expenditure of those resources. In fact, it may be thought of as irrational to do otherwise.
A trivial example of this can be seen in a case where person A repeatedly introduces facts which have little to no impact on the considerations of person B (Did you know the Bullet Ant is 1.5 inches long? Did you know the aurora is caused by charged particles? Did you know Yoda's style of lightsaber combat is called Ataru?). The failure to prioritize the communication of facts and opinions and to stymme wasteful communication could be quite limiting in the face of one who raises facts such as this.
If anything, the only mistake in the phrasing is that person B should state that "they believe there are more important things to talk about." As you wrote it, their phrasing suggests they know what is valuable for both parties, but the slightly modified phrasing permits them to communicate their opinion and save both parties from wasteful communication (its painful to finish a logical argument only to find the other person had found other things to do besides listening). A continued discussion as to why they believe this may bring agreement between Person A and Person B, or it may identify disagreement as to the validity of Person B's belief.
This could be an outright fallacy if Person B cannot identify anything more important to talk about, from their point of view. There may also be a fallacy in their defense of their belief that there is something more to talk about, but that fallacy is in the (unspecified) defense, not in the retaliation itself.
In some situations such a response could be considered rude, but hardly a logical fallacy.