What is a borderline case, and its relationship with vagueness? I have googled a lot but didn't quite find anything that suits my understanding.
Best not to overthink this. A borderline case is merely a case that partially, incompletely, or imperfectly fits a category, theory, or paradigm. For instance, if we think about the category 'motor vehicle' there are the obvious cases like cars, motorcycles, and trucks. But then there are an assortment of things like electric scooters, motor-assisted bicycles, Segues, and hoverboards that are technically 'vehicles with motors' but are not normally thought of as 'motor vehicles'.
Sometimes borderline cases are mere annoyances of the "I don't know what to do with this" variety; sometimes they provide fruitful areas of research or point out places where definitions need to be tightened or refined. In any case, they are pragmatically unavoidable. Every human concept inevitably has a clear center with fuzzy edges.
A lot of thinkers fall into a trap of thinking that there is one correct definition, and that somehow language is tidy, like a mathematical theorem. No such luck. Ludwig Wittgenstein famously adduced the family resemblance argument during the linguistic turn to push back on the idea that words have a purely objective status. Today, some linguists defend prototype theory extending from there philosophy of language to show that there are borderline cases, cases that don't neatly fit into a category.
The example I like to use is the spork, and the question, is the spork a spoon or a fork? The claim that it's spoon-like and fork-like appeals to our philosophical intuitions. But if you had to put away in your silverware drawer, do you put it with the spoons or the forks? I think it's fair to say, as trivial as the question is, it raises the same sorts of questions as the sorites paradox. Borderline cases, or any of the synonyms, are thus challenges to our thinking philosophically.
To be technical, some entities have a mix of properties that challenge our categories. These entities then challenge our notions of identity, membership, and analogy, and even our presuppositions about the Laws of Thought. Famously, Victorians tried to argue natural kinds in a last ditch effort to sustain Aristotelian thought. Today, biological notions of clades take a more relativistic approach to categories. A platypus is another example of a borderline case. Borderline cases therefore go to the heart of profound metaphysical debates, such as the validity of the debate between scientific realism and instrumentalism.