I will distinguish two parts of this question. The first question is, do the examples you presented show that mathematics is arbitrary. Second is, whether there can be alternative mathematics in such a way that choosing between the actual mathematics and the alternative is an arbitrary choice. I’ll start by addressing the first one.
Switching the definitions of odd and even merely changes the words that we use to refer to the properties of numbers. So the property of being odd would still be the same property regardless of what we want to call it. Of course, the choice of words is not entirely arbitrary, since it is at least in some sense conventional. What I mean by that is that the definition “Number is even if and only if it is some equal to two times some other number” corresponds with the English meaning of the word “even”, so it is advisable to use this definition, since it makes reading theorems easier. Of course, not all definable properties of numbers correspond neatly with an expression in English, but even in these cases, there are practical considerations to take into account when choosing a word to refer to that property.
Even if you would want to maintain that the choice of words is arbitrary, this would still not show that mathematics is arbitrary since mathematics is about numbers (and sets, functions, lines, spaces etc.) and not about the words we use to refer to numbers. Like Shakespeare once wrote: “Even number by any other name is still devisable by two.”
What about the case of “let n be an arbitrary integer”? This is, in fact, a valid inferential step, but I feel it is not usually explained very well during introductory courses which can be a source of confusion. The rationale behind this expression can be demonstrated this way. To show that no card has a number on it greater than 10 you can pick a card from a standard deck. Now without turning the card over you know that it is either number card or a face card. If it is a face card there is no number on it. If it is a number card it is the highest value is 10 which is less than 11. By the transitivity of less-than relation, the statement holds for all cards with the value less than 10. Now you probably realize that there is no longer any point of turning the card over since whatever it is it the statement holds for it. Therefore, one can infer that the statement holds for all the cards.
Inferences made with arbitrary integers are analogous to the above example. There are a lot of things that can be known about n just from the fact that n is an integer. Arbitrariness here just means that we don’t assume it to be any specific integer and this allows us to make universal claims about all integers. This does not show that these inferences are arbitrary or that mathematics is arbitrary.
To turn to the second question, is mathematics and/or logic is arbitrary? I take it that you mean that since in mathematics some sentences are called axioms and from those, we derive other sentences with some rules of inference, could we then just change the axioms or rules of inference arbitrarily? This sort of idea is usually associated with formalism in the philosophy of mathematics. While it enjoyed some popularity during the beginning of the 20th century it is nowadays considered to be problematic. I’ll sketch some objection typical raised against it.
The first objection is if any axioms will do, why we are so interested in certain axioms like axioms of arithmetic or geometry? What makes these axioms special and axioms that talk about centaurs and unicorns uninteresting? The second objection is that, at least to many mathematicians, axioms like “Successor of a natural number is a natural number” seem to be true and obvious. (How we can know that they are true is problematic, but axioms obviousness is somewhat hard to explain if the choice of its truth value is arbitrary.)
The third objection has to do with the application of mathematics. How can arbitrary set of sentences and rules of inference produce verifiable predictions about concrete objects? If I put 7 marbles to a bag that contains 5 marbles is it really a coincidence that the bag now contains 12 marbles?
Also depending on precise formulation, formalist may need to deal with some technical objections such as those relating to incompleteness theorems.
This is by no means a refutation of the idea, nor a comprehensive presentation on the subject. For further reading, I’d suggest these articles from Stanford Encyclopedia of Philosophy. They are comprehensive, but a bit difficult.
If you are interested in more introductory style text, then Steward Shapiro’s Thinking About Mathematics has a chapter on formalism and deductivism.
Rudolf Carnap's views on conventionality of mathematics are summed up nicely in “Empiricism Semantics and Ontology”. (Quine’s critique of Carnap's position can be found in “Truth by Convention” and “The Two Dogmas of Empiricism”.) These may give interesting pointers about arbitrariness even though I would distinguish conventionality from arbitrariness.