For the specific software situation you're describing, take a look at Hoare Logic, as discussed, e.g., in https://en.wikipedia.org/wiki/Hoare_logic
In this model, your program's correctness is determined by a collection of Hoare Triples of the form {P}C{Q}, where {P}=preconditions/input, C=your_program, {Q}=postconditions/output. Given input satisfying any of the prescribed preconditions, if your program's output satisfies the corresponding postconditions, then it's "correct".
As a simple example, suppose C=program_to_add_two_numbers. Then one Hoare Triple might be {P}C{Q}, where {P}=both_numbers_to_be_added_are_odd, and {Q}=sum_is_even. Obviously, that one Hoare Triple by itself doesn't establish C's correctness. But an appropriate collection of Hoare Triples usually does. ("Usually" means whenever there exists a collection of Hoare Triples that uniquely, in an if-and-only-if sense, specifies the function : input-->output to be calculated.)
And you might want to check out some of the "See Also" links at the bottom of that wikipedia article for several other models of program correctness. As you might imagine, computer science has devoted lots of rigorous investigation to this subject.
edit: Thanks, @JamesKingsbery Regarding your "term for this concept" remark, yes, I should have explicitly mentioned that you can google the term "program correctness" for lots of additional information.