If by "experience" you just mean qualia then the "connection" is purely accidental. We do not need to compare qualia to agree to call the same objects "green", nor do we need it to associate them to certain frequencies or spectra. We can equally well agree on outputs of optical spectrometers without that. By the way, most colors are not associated to a single frequency, they are combinations of monochromatic waves with a spectrum of frequencies, but mostly well approximated by weighted sums of the "primary colors", RGB, to which our retinal cones are specifically attuned (at least on the traditional trichromatic theory, it is contested). There are some more subtle things that we can compare, like whether or not we agree that some shade of blue is "between" the other two, this is how shades are correlated to spectra more specifically and the psychological color space is mapped out, see Bimler's Psychological Color Space and Color Terms. But this is still relational information about arranging the shades relative to each other, it conveys nothing about what the shades "feel like".
In fact, it is unclear if "knowing" someone else's qualia is a meaningful idea, and hence if the question of having the "same" or "different" qualia makes sense at all. You can not see my qualia because for that you'd have to become me and then "you" won't be seeing them, and if they somehow become "yours" they will no longer be "mine". Such are the rules of qualia. We might speculate about some "common nature" of the human race, which ensures that our qualia "truly are" the same. But again, the question is not even whether this is true, but what possible sense it can make.
And we do know that the supposed "common nature" is not that common. Indeed, we can attach some meaning to "seeing colors differently" because different people differ on color even relationally, e.g. color blind people do not distinguish shades that the majority does. Dalton describes his color blindness as follows, see Is it possible that I see color differently?
"That part of the image which others call red appears to me little more than a shade or defect of light. After that the orange, yellow and green seem one color which descends pretty uniformly from an intense to a rare yellow, making what I should call different shades of yellow”.
This should not be surprising. As Poincare observed in The Value of Science, science, being objective (by which he meant intersubjective, communicable), can study nothing about "qualities" of things in themselves, only about relations among them:
"Sensations are therefore intransmissible, or rather all that is pure quality in them is intransmissible and forever impenetrable. But it is not the same with
relations between these sensations. From this point of view, all that is objective is devoid of all quality and is only pure relation... we must nevertheless admit that nothing is objective which is not transmissible, and consequently that the relations between the sensations can alone have an objective value."
Eventually this position evolved into structural realism, the most popular kind of realism today, see Brading's Epistemic Structural Realism
and Poincare's Philosophy of Science. Many dispute that even relations (or relations among relations) are communicable due to indeterminacy of translation and other reasons. But qualia are the first to go, and at best the frequencies reflect relations among qualia only.