What sorts of restrictions, if any, would be needed to ensure that a new operator "plays nice" with existing operators? That is, you could combine them freely and not cause any problems or inconsistencies?
Strictly speaking, when you are talking about designing any sort of formal logic, you typically are expressing an aspect of natural language that the designer wants to express. For instance, modal logic expresses epistemic modality present in language, sentential logic captures the logic that governs the interaction of propositions given classical logical presumptions, predicate logic the logic of relationships in sentences in phrasal syntax, etc. Therefore, the constraints placed on a logic are teleological in nature.
What is important to note is that logics conform to a grammar in the way that formal languages and automata define them, as a series of transformation rules that move from initial symbols to terminating symbols using an alphabet in a formal system, and the transformational power of any logic conforms to the general mechanics of a lambda calculus which involves three acts: transformation, definition, and application (the three axioms of the calculus). So, the constraints on a logic must include compliance with the general form of any logic as expressed by the axioms of the lambda calculus.
But the answer to your question isn't found in the grammar of the logics proper, rather the goals of the designer of the grammar. When George Boole created Boolean logic in 1854, he was trying to create a grammar that caught truth conditional semantics that captured the essence of the Laws of Thought. Brouwer's constructivist logic sought to weaken those constraints by modeling natural language use that ignores LEM and DN, a fact that allowed Martin-Löf to extend Russell's simple type theory. And in fact, "avoiding inconsistencies" in terms of truth-semantics itself isn't even necessarily something one has to do when one designs a logic, since paraconsistent logic shows that a logic can capture the essence of logical inconsistency itself to show how a language can tolerate contradiction.
Therefore, the design of a logic and the playing nice of operators is really is a function of the intent of the designer, and the transformation rules abstracted from natural language incorporated into the logic through grammatical codification which are subject to the whim and utility of the inventor of the logic. Historically, the constraints were to show how propositions and then predicates related according to the Laws of thought, but logics have moved to embracing more sophisticated semantics that we find in natural language, a fact noted by Montague post-UG when he built his grammar which attempted to capture the essence of "material logic" which has been borrowed by software engineers in their more sophisticated NLP systems.
EDIT for explication of the conclusion of the first half:
A "general sentence operator" is just a syntactic unit in a formal grammar to express an action semantics that inheres to natural language. If you are looking for understanding how to reconcile two operators, the constraints are in the coherence of the action semantics, not in the syntax, and the semantics exist in a possible world as extensive as the natural language semanatics which exist in the mind of the engineer of the logic.
In other words, as PW_426 notes "There isn’t a general procedure to determine the restrictions required for combining certain operators." The operator is an expression of the logic that inheres to the metalinguistic description of the logic system as expressed in natural language. Therefore to get two operators to "play nice" merely means that they express a coherent semantics in the metalinguistic framework of the logic, the... this-is-what-those-symbols-mean-ness.
Exemplification is clarification. Consider the box operator. What is it's relation to a negation operator? How do they play nice in regards to being a constituent or a modifier of a proposition. It has nothing to do with the formal logic, and everything to do with the natural language semantics of "necessity" and "negation". In fact, they are distinct concepts, therefore they play nicely because they are complementary. The inclusion of a diamond operator in a logic dovetails nicely with the box operator because "possibility" and "necessity" have a relationship in natural language.
What is implicit here is that all operators in all formal languages express an aspect of the logic that inheres in natural language using the formal grammar as an abstraction, and the lambda calculus as the rules for the formal system. Coherence of "general sentence operators" exists as a function of the coherenece of the natural language description of them, and is not subject to objective and external constraints related to the syntax or the logic proper.