This is my position, which could conceivably be wrong. I am not aware of any unanswered criticisms of it.
There is no infinite regress because justification is impossible, unnecessary and undesirable. If you assess ideas using argument then the arguments have premises and rules of inference and the result of the argument may not be true (or probably true) if the premises and rules of inference are false. You might try to solve this by coming up with a new argument that proves the premises and rules of inference but then you have the same problem with those premises and rules of inference. You might say that some stuff is indubitably true (or probably true), and you can use that as a foundation. But that just means you have cut off a possible avenue of intellectual progress since the foundation can't be explained in terms of anything deeper. And in any case there is nothing that can fill that role. Sense experience won't work since you can misinterpret information from your sense organs, e.g. - optical illusions. Sense organs also fail to record lots of stuff that does exist, e.g. - neutrinos. Scientific instruments aren't infallible either since you can make mistakes in setting them up, in interpreting information from them and so on.
What about Klein's specific argument? This is given here:
http://www.arts.cornell.edu/cag2/papers/Infinitismdebate.pdf
He assumes that justification is possible, necessary and desirable. He then argues that other accounts of justification don't work and that the best objections against infinitism don't work. The objections he addresses are
(1) You can't do an infinite number of steps. Klein claims that what matters is that there is no proposition that couldn't be justified, even if it isn't actually justified.
(2) If there is such an infinite stack of propositions then at some point they will be so complex that no finite mind can grasp them. Klein claims that the propositions need not increase in complexity in this way.
In reality, Klein's position is not tenable and this is not primarily because you can't make an infinite number of justifications, but, rather, because even the first justification doesn't work.
We don't create knowledge (useful or explanatory information) by showing stuff is true or probably true for reasons so how do we create knowledge? We can only create knowledge by finding mistakes in our current ideas and correcting them piecemeal. You notice a problem with your current ideas, propose solutions, criticise the solutions until only one is left and then find a new problem. We shouldn't say that a theory is false because it hasn't been proven because this applies to all theories. Rather, we should look at what problems it aims to solve and ask whether it solves them. We should look at whether it is compatible with other current knowledge and if not try to figure out the best solution. Should the new idea be discarded or the old idea or can some variant of both solve the problem?
See See "Realism and the Aim of Science" by Karl Popper, especially chapter I and "The Retreat to Commitment" by W. W. Bartley III.