It seems to me the only thing about colors that can be understood rationally is that they are distinct. We interact with other people who for the most part label my red as their red, my green as their green, and so on. It gets more complicate with scientific knowledge. We can interact with scientific instruments and measure the wavelengths of light and distinguish between colors. The point is that we don't know anything more about the subjective experience of color for ourselves or for others. We still have the question, is it possible that my red as I subjectively experience it is another's green, and their green is my red? (This question is discussed in this Vsauce video.) It is entirely possible since the only necessary thing is that I can distinguish between (at least some) colors in order to function in the world. If I was color blind, I presume there is some distinction, if limited, to be made. But there is still this issue of subjective experience.
Now suppose solely a material world exists. Then a simplified universe would be akin to a bunch of floating marbles in a jar, evolving in time and perhaps in complex patterns. There is the question of why there are any marbles in a jar to begin with, of which it is probably impossible to give an account as a materialist since it is a fundamental belief. But it seems to me there is an additional component of the subjective experience of color beyond "the marbles". If "seeing colors" is marbles being able to avoid all the walls in the jar, then observationally we see marbles making distinctions. But in a sense I am a collection of marbles, and I seem to experience something beyond this (seeing colors) that cannot in anyway be truly correlated with or explained by the organization of the marbles.
Does this give evidence against materialism? How do materialists reconcile this issue? This most certainly touches on the issue of consciousness which one might say causes problems for materialists, but I would like this specific issue of colors addressed.