Of course, it's legitimate to be philosophically uncommitted about the reality of an infinity of material objects, until a proof is given one way or another. And given that all ontological arguments are suspect, I strongly doubt the possibility of a pure philosophical argument for an infinity of things. There may be scientific arguments for this, which can be believed with more or less degree of certainty. And one might even be able to provide an argument that the concept of an infinity of material objects is inherently inconsistent. That wouldn't need to be an ontological argument, so I suppose such an argument could be attempted, but I doubt there are any good arguments like that.
So from a purely philosophical perspective, I don't think you can make a decision about the real world. From a scientific perspective I think both possibilities are viable.
Now you mention being particularly interested in there being no infinity because we cannot know an infinity of things--but you seem to base this on the assumption that there is no material, or physical, or ontological infinity. But as I've described above, I don't think that's a philosophically proven thesis, so the foundation of that argument may not be sound. But perhaps I misunderstand you when you say "deny that we can know an infinite amount of things ... since ... there are only finitely many".