But it would seem that the origin of our language is rooted in
ostensive definition. It is after all how children with no prior
knowledge of a public language begin to learn a language. Their mother
or father points to something while stressing a given word.
That would be Augustine's reasoning (or rather, remembrance) that Wittgenstein criticises. But it is hard to agree with Augustine. Children definitely do not learn their first language by the method you suggest, which would give them a lexicon, but no grammar.
I have seen little children babbling, imitating adults in conversation. Their sounds make no sense at all, but what is important is that they have grasped that talking is important, because adults talk. Meaning comes afterward, and it is only secondarily linked to a list of substantive nouns that children learn by heart. Instead, children must first learn to do basic operations such as demand, refuse, accept, call, react to calls, etc. They do not learn to call their mothers "mummy" by someone pointing to their mother and calling her "mummy". They learn it by realizing that their mother reacts immediately and joyfully if they emit sounds like "ma", "mo", "mem", "muma", etc.
As such, what allowed him to criticize ostensive definitions in the
private language argument while not coming to criticize language
acquisition through ostensive definition in the case of children?
As far as I understand, he opens Philosophical Investigations by exactly criticizing language acquisition by ostensive definition in the case of children. In fact, it seems to me that he strawmans a remark by Augustine that has no greater ambitions than recalling a first childhood that has actually been forgotten into a whole "philosophical theory of language" that doesn't seem to match any actual body of philosophy.