This is not about philosophy as much as psychology. (But our systematically skewed perception of social assessments is something philosophers should keep in mind. If someone could formulate it well, there is an interesting question in why the stuff most important to us is most often perceived least objectively.)
Estimated via serotonin and cortisol levels, confidence actually predicts competence pretty well in the median-earning range of the population. One theory is that this is the range for whom feedback about competence, especially from our education system, actually bears on self-image, while people of privilege and people of either very low or high ability are often more immune to feedback. And those latter are the people we think of when informally estimating social effects.
A history of success predicts future success, and also predicts confidence, up to a point. One theory (from studies of gifted children) is that this is the point where more acute awareness in general may prime more acute self-awareness and lead one to over-correct, (and the curse noted by Bertrand Russel quoted in the comments begins.)
For instance on the up-side, IQ predicts both confidence and future income, up to a certain level, and then it stops. Most people's IQ's lie well below that cutoff, but the most noticeable people's lie above it. Something similar happens on the down-side -- e.g. the dumb and the arrogant are both more likely to go to jail, but the bet is on the dumb over the arrogant, while the arrogant are the more attention-grabbing cases. So our perception of how much these traits are correlated is skewed.
We overestimate the limitations on this effect, because we are focused on extreme cases, but the mode lies near the median, so we are misled by that impulse. Given that flaw in our perception, we should look at a couple more concrete observations that apply more directly:
The feeling of being powerful has been experimentally verified to allow one to think more abstractly more often, which pays off in modern life. When mixed with raw competence, the tendency to frame things abstractly allows for future improvement, in a way that simple hard work does not. So, to some degree, especially in the young, confidence predicts potential future increases in competence.
Also, risk taking is necessary to leverage competence for actual achievement. Confident people, especially males, are valued for their ability to face risk productively in situations where more competent people might be put off. In a group setting, you often need one party's confidence in order to adequately deploy another's competence -- we are all familiar with the strange stereotype of the underling who actually does all of the work that creates the reputation of 'the great man'.