Utilitarianism has the unfortunate position of being an excellent example of the Münchhausen trilemma. This is a skeptic's issue with any system where you try to answer "how do I know this is true." It makes the claim that any theory which must prove that it is true will run into one of three patterns:
- It will enter a circular argument, where the proof supports the theory and the theory supports the proof.
- It will enter a regressive argument, where the proof involves an ad infinitum argument (such as mathematical induction)
- It will be axiomatic, requiring an assumption that is not proven.
Hans Albert, the originator of the trilemma argued that every theory which required a proof would have to fall into one of these. This argument stands as a challenge to the concept of justification in the theory of knowledge, and has appeared in many forms since as long ago as the Greeks (Aggripa, from the 1st century BC).
By this argument, we should expect every ethical system which attempts to justify itself to run into one of these three tropes, and indeed we do. As a vague generalization, most ethics systems are rooted in axioms, which generate their own challenges. Utilitarianism does not explicitly stand on the axioms. It, instead, stares the trilemma straight in the face and says "sure, justification may enter one of those three states, but we're not going to be specific about which one!"
Accordingly, some Utilitarian arguments assume there is some axiomatic definition of utility, and go to make claims about what ethics must come from such a definition. Others enter regressive arguments, having utilitarian systems which model other more nuanced utilitarian systems, ad infinitum, with the claim that there is no system of ethics which cannot be modeled as an infinite regression of utilitarian systems. Some Utilitarians, as you mention, embrace a circular argument, where it is argued that the best way to determine what is ethical and what is not is to use a utilitarian ethics system as a tool to learn more about your own ethics.
This refusal to fall into any one easy category puts Utilitarianism in an interesting position. Proponents of it dance around any one trope from the trilemma, demonstrating how it doesn't have to fall into any particular pitfall, while opponents intentionally pigeonhole it into the trope which they find easiest to argue against.
If you wish utlitarianism to be built off of an existing ethics system, you simply need to define a measurable metric to determine the utility of any ethic. On the other hand, if you want it to bootstrap itself, you might use it as a tool alongside Baysean Inferrence to identify arbitrarily better models of your ethics, until one day you find that the results of the model are indistingushable from your system. There are utilitarian systems which assume complex metrics for each individual and super-simple ways to aggregate utility for a group (such as "maximize the average utility"). There are utilitarian systems which assume simple metrics for each individual, but combine them in very nuanced ways to generate gestalt behavior in a group. Its flexibility can be its own undoing.