If I ask the question: "Is it possible to define knowledge by proving that there exists something that can't be known?" to which branch of philosophy does this question belong?
Is it transcendental philosophy?
Answer: It depends, or doesn't. First, since your question concerns knowledge and "how things are known" it belongs to epistemology.
It does appear to be a fair example of "transcendental" philosophy, as originally developed by Kant. Such philosophy addresses not only analytical or "rational" truths and "empirical" knowledge from experience, but knowledge about "possible experience." It concerns knowledge outside of experience (hence "transcendental"), yet inferred from the a priori structure of concepts that makes experience itself possible.
However, Kant's transcendental philosophy would not admit knowledge that in principle "can't be known." Like Locke, he does admit that, yes, there always remains "something" that we do not and can never know. Kant called this "noumena" or "things-in-themselves" existing independently of anything we could ever know about them. Locke called this simply the "I-know-not-what" behind our perceptions. By definition they are not "known," for we can say nothing about them. Critics of Kant argued that this was incoherent. How could Kant then "know" they exist and "say" or "affirm" that they exist?
So your question appears to be in this camp. However, this is far trickier than it seems. (Alas, nothing about transcendental philosophy is simple.) Your question could also be simply analytical. This is more difficult to unpack. Your question could be answered by an analysis of the terms with no recourse to "possible experiences." It isn't always easy to tell and can take some work.
So, it depends on how you develop your answer. You could pursue a "transcendental deduction" dependent in principle on possible experience, Kant's "synthetic a priori." Or you could pursue an answer by logical analysis of the terms. Your use of the word "define" does suggest the latter. Good hunting!