OK, I'm going to have a go at this.
An argument can go 3 ways:
- A circular path.
- Infinite regress.
- Hmm... let's just call it stop condition for now.

Circular arguments
These are outright nonsense. A thing cannot prove itself.
Oil floats on water because oil floats on water.
Nonsense. The premises are right, but the argument is nonsense. Compare with:
Oil sinks in water because oil sinks in water.
Takeaway: for an argument to be reasonable, the premises have to be reasonable, and the argument valid.
Infinite regression
Let me give you an argument to tinker of:
For each number X, there exists a number Y, such that Y = X + 1.
Try to stop that regression train.
"Each and every single human being was created by a human being before them"
This is an infinite regression alright, but it's nonsense as well - humans don't create other humans. Biological processes 'do'. We are merely hosts for a biological phenomena.
You have to be careful when using the word create. Does a carpenter create chairs? Language is fuzzy and abstract - a carpenter merely converts one form to another. The end result can be seen as having been 'created', but really, its just about cutting wood and assembling it in a particular way.
Now you'd be a famous person had you managed to prove that infinite regress is impossible. For all we know, perhaps the most empirically tested paradigm we have is cause and effect - break it once and science will be reduced to a pile of nonsense.
All these talks about initial singularity are based on general relativity, which is by many seen as fit to explain the early universe as Newtonian physics was explaining the orbit of mercury (both are phenomenal theories of unprecedented utility, but neither is all-encompassing).
Notwithstanding, lets play along - lets just assume that infinite regress is nonsense.
Stop condition
A stop condition can be many things. An axiom or justified belief, for example.
Now stop conditions are not evil. In fact, they are highly practical. Consider this:
The amputated limb of a human will not regrow.
Why?
Because no such case has ever been observed.
(The skeptics will surely have their say on this, but this is manageable alright.)
The commutative low for addition is another axiom that serves us right.
Anyhow, with circular reasoning being nonsense, and infinite regress out of the question, all you have shown is that arguments must stop at some stop condition. Let's put this to the test:
There is at least one day a year where not a single person on this planet is having sex. Why? Because god exists.
OK, we have a stop condition, it's even god itself. But is the argument valid? Is it reasonable?
Now consider this argument:
Given that circular reasoning and infinite regress is nonsense, there must be a stop condition.
OK. Seems like a reasonable argument, but note that this in itself depends on an axiom - that there are only 3 ways an argument can go. But lets assume the 3 way axiom is reasonable.
Now you argue:
Given that infinite regress is nonsense, god must exist.
If this argument is valid, so is this:
Given that infinite regress is nonsense, love is red.
You have asserted that:
God is only the first cause.
And so one might argue:
A love that is red is only the first cause.
It's a bit unfair to take a word like god, the semantic of which is fairly understood, and demote it to something that only constitutes a first cause. What's the point doing this? The concept already has a title - first cause.
Imagine I walk in the street and this girl walks to me and say:
Hey, god exists.
And I go, really? And then she goes:
Yes, but its not the god you think of, it just another label for the first cause. God is just another label.
It's 3AM, so I'm unsure how much sense this makes. Probably not much. But I hope this helps.
Possibility vs practicality
In light of your comments, I'd like emphasis a point that so far may have been implicit.
I believe your line of reasoning is this: If we'd have to regress ad infinitum, we'd get nowhere. This is absolutely right.
To make it as bold as I can, knowledge depends on stop-conditions. Euclid's proof of the infinitude of primes is a proof alright, but only for being based on some mathematical axioms, which are its stop condition. We must stop somewhere on our way in, in order to get results back out.
But this doesn't make infinite regress impossible, it is simply impractical.
Alice
By way of analogy, consider you walk in the woods and suddenly see a door. You open it and see stairs. A white rabbit pops from the bushes and says:
Howdy! These stairs go down forever. You can go down if you fancy, but they never end.
Now. Would going down the stairs be impossible or impractical? If the stairs are indeed infinite, would it be impossible for a thrown ball to fall down infinitely?

Impossibility
If it is impossibility we debate, then cause and effect couldn't serve as a better example - we have all the evidence in the world such principle is in effect. We haven't got a single evidence that there is an effect without a cause. In other words, infinite regress seems much more reasonable (possible) than any 'natural' stop condition.
The same principle is in effect elsewhere (forgive a few leaps):
- We are having this discussion using a language.
- We have acquired it through experience.
- An experience which is the results social formation of shared abstractions.
- Society itself was shaped by evolution.
- Which takes us back to the big bang, which is currently as far as we can empirically test our knowledge. The big bang is forced upon us at the moment same as the notion of the sun orbiting earth was forced upon our ancestors before they had the tools to conclude otherwise (in other words, before further regression was possible).
On the nature of science
As a closing remark, I hope we have a shared understanding that science is not concerned with deriving at absolute and independent truths - science is about adopting the most rigorous tools at our disposal to approximate patterns.