The circularity problem is a flaw in all variants of justificationism: the idea that it is possible or desirable to show your ideas are true or probably true. This is the standard most philosophers have adopted for epistemology. Quine does not explicitly reject justificationism, so his ideas have been interpreted in the light of standard justificationist ideas and can be criticised in the same way. In reality, you can't prove any position or show it is probable. Any argument requires premises and rules of inference and it doesn't prove (or make probable) those premises or rules of inference. If you're going to say they're self evident then you are acting in a dogmatic manner that will prevent you from spotting some mistakes. If you don't say they are self evident then you would have to prove those premises and rules of inference by another argument that would bring up a similar problem with respect to its premises and rules of inference. Most philosophers have ideas that are subject to exactly the same criticism so they are partners in crime with Quine.
All knowledge is created by conjecture and criticism. You notice a problem with your current ideas, propose solutions, criticise the solutions until only one is left and then find a new problem, see Chapter I of "Realism and the Aim of Science" by Karl Popper. There is no circularity in real knowledge creation because real knowledge is not justified.
There is a more pressing problem pointed out by Popper in "Logic of Scientific Discovery" Chapter 2. (This was written before Quine's stuff on naturalized epistemology and published in English in 1959, but Quine doesn't seem to have noticed it.) Epistemology is about methodology not what people actually do, nor can it be. The idea that you can get epistemology from just observing what people do is uncritical. It does not admit the possibility of improving on what scientists currently do since any improvement would not be the same as what we have observed already. Also, how are you to decide whether a person is a scientist in the first place if not by having rules about how a person ought to behave if he wants to discover how the world works and comparing what a given person does with that standard?