Is there a branch of philosophy that deals with the concept of classification? I guess that perhaps it's metaphysics or ontology - they are the nearest things I can think of, but I'm not really sure.
Assuming you mean the problem of classification in general, i.e. placing entities with various qualities into meaningful relations or subgroups on the basis of those qualities:
Extensive experience with machine learning has demonstrated pretty conclusively that we don't/didn't know from first principles what is necessary for useful classification. This is not to say that there is not work done on the philosophy of classification; there is, resulting in debates including, for instance, one about the periodic table of the elements.
Unfortunately, when it comes to actually building a classifier, insight from philosophy has not proven adequate. Progress, such as there is, seems to be coming from statistics and mathematics (especially but not exclusively Bayesian statistics). I think the jury is still out on how to make a robust general classifier and thus how to draw meaningful and robust distinctions; once we've got something that actually works (and works well), I think we'll be in a good position to reason about the principles that make it work.
(For reference, some good sounding frameworks that don't work alone include trees and other forms of hierarchical clustering (because every concept must have only one parent, while real objects like a bobble-head Obama toy fall squarely into multiple categories, and picking one as the "real" parent idea makes lookup and reasoning hard when you start with a different parent); tables of attributes (because it is difficult to restrict the number of attributes to a finite list which is nonetheless useful, and it's hard to work with such tables once you've got them); and networks of associations, oddly sometimes called rhizomes in philsophy despite the concept being around far longer in machine learning research (because we don't know how to build them in such a way that they're easy to work with yet capture all the subtleties and quantitative aspects of the world). All of these are surely relevant and useful in at least some subdomains.)
The particular classification science you are looking for when elaborating tree-like structures is cladistics:
it is an approach to classification in which units are grouped together based on whether or not they have one or more shared unique characteristics that come from the group's last common ancestor and are not present in more distant ancestors. Therefore, members of the same group are thought to share a common history and are considered to be more closely related. Its born out of taxonomy. The determination of what counts as a unit here is a species which is properly & fruitfully designated as those organisms that can inter-breed with each other.
However, additionally, perhaps the french philosophe Deleuze thinking may be helpful here.
He draws a distinction between aborescent & rhizomatic thinking. That is tree-like and network-like.
As you observe tree-like classifications are everywhere in different scientia, but this tends to be because of the simplicity of the description. That is the tree is neither too simple or too complex that it is almost useless as as classification. For example, one can plot the sciences on a line such as: biology -> chemistry -> physics; this of course captures one important dimension of scientific ontology, but misses out many others. For example the human sciences do not even appear on this classification line. Such simplicity leads to caricature. There being no dimension of time doesn't show the budding of new sciences, nor loss or ossification of old sciences - what happened to alchemy? That is intensities are missing.
The tree of life determined by taxology determines how species are descended from each other. Its modern incarnation is cladistics which uses gene-determined information.
Now, looking at the traditional classification of languages, philologists have determined several trees. The Indo-European, Semetic or Sinitic language families. But of course there is word adoption, algebra from al-jabr, sin from sinus etc; meaning drift such as gay as in happy to gay as in queer. That is again intensities, connections, divisions and history are missing.
The rhizomatic picture is a philosophical concept devised to fix this obsession with trees. To stop thinking tree right out-of-the-box and think out-of-the-tree and towards some new picture.
This picture opens up new horizons wheras the tree anchors - for example the archaic controversy as to whether Indo-European languages has an originary point is plausibly driven by the very form of the picture of a tree; one might instead theorize multiple diffuse sites, flows & mutations.
One might suppose with the greater computational & data technologies at hand one needs a more sophisticated understanding of classification that elaborates and constructs a typology from the point (no classification) to the rhizome (all significant determinations & dimensions captured). Both these ends being ideal ends - there being no point to the point - and the rhizome always escaping.
The concept of classification seems obvious first, in the sense that you recognize a classification when you see one, especially if it's a nice classification. Even nice classifications leave enough room for philosophical questions, and ontology deals with some of these. However, sometimes no nice classification exists, and just completely abadoning established classification principles and going with something completely unrelated like rhizomes instead doesn't really address the concept of classification.
One may look for mathematical branches instead, and claim for example that lattice and order theory subsume the concept of classification. However, this would ignore the fact that the actually published results focus on mathematically interesting problems instead of investigating practical questions related to classifications. Approaches like formal concept analysis try to restructure such mathematical developments with philosophical notions like extension and intension.
FCA aims at the clarity of concepts according to Charles S. Peirce's pragmatic maxim by unfolding observable, elementary properties of the subsumed objects.
Those approaches fight with problems like their perceived triviality, with getting reinvented multiple times with minor variations under different names, with the "ugly" classifications you get in cases where no nice classification exists, and the non-triviality of appropriate efficient mathematical algorithms. But those approaches get many basic things right:
- The initial information is represented as a bipartite graph. This is good, because this representation is still unstructured, and the underlying order is only implicitly present.
- The concept-lattice is represented as two complementary semi-lattices (in the form of set algebras).
- As Christopher Alexander noted in "A city is not a tree", semi-lattices generalize trees and avoid their main drawbacks.
- This is a very appropriate representation of hierarchy as it actually occurs both in mathematical structures and in general classifications of knowledge.
- This structure of a Galois connection is typical in many contexts.
- In a tree, it's easy to focus (zoom-in) on a sub-tree, but it's much more difficult to see the more general context (zoom-out). A semi-lattice would be similar to a tree in this respect, but the structure of a Galois connection allows to both focus and contextualize easily, by using the complementary representation.