You are asking a question with a bit of subtlety here. How much of mathematics consists of mathematical writing — that is, general attempts by the author to convey an idea (if only to themselves in the future)? And of course, what do you mean by art?
Can Newtons Principia, for example, be considered an art object; or Einsteins papers?
Undoubtedly yes, particularly in the case of Einstein's work. His famed Gedankenexperiments, and the logical consequences thereof, are the subject of this work of art; his prose is the execution — which are the more excellent because they don't involve heavy mathematics; his use of simple algebra to tackle the subject of what we call special relativity is routinely extolled by physics educators, and amounts to an appraisal of artistic technique used to convey an important subject. That the subject is important intellectually and instrumentally rather than grasping at human emotion directly is, to me, beside the point. Furthermore, as an afficionado of clear communication, I would rate the clarity of the prose as nearly matching Bertrand Russell, another great mathematician and (quite independently but more importantly in this case) explicator, whose prose is a pleasure to read because he makes approaching subtle distinctions nearly effortless through his precise but flowing style.
I am less directly familiar with Newton's Principia as a body of text, and perhaps a modern appraisal would find it wanting somewhat in style; as medieval artists today might be criticized anachronistically for their lack of use of perspective (for realistic geometric representation, not only being unavailable to them, was perhaps not their intent either). Newton uses a lot more argumentation using Euclidean axiomatic constructions than any modern approach would ever use; but of course, Newton invented most of the tools we would use today to convey the subject, and was secretive enough not to use them in his own treatise on natural philosophy. We might construe Newton's work as an exercise in constraint as a body of art treating its proper subject, but it is a body of art that would be extremely challenging to most who tried to appreciate it.
But these are the works of masters, and besides are mostly full of prose. You probably mean to ask whether we are meant to interpret the formal symbolic treatment of the mathematics as poetry. Calling it "poetry" is a bit much, perhaps; but it does get across that I mean a style of writing which is in sharp distinction from representing anything like normal speech. Indeed, writing a piece of mathematics involves a lot of precise repetitive acts and carefully judged movements in particular directions, so that if one must describe an analogue to mathematics in another creative field, it would be perhaps more appropriate to compare it to dancing, and in particular ballet. Certainly I am struck by the extent to which a mathematical development can be graceful and flows smoothly — or clumsily fails to do so.
If mathematics is an art, is it a representational art? Absolutely, yes. Even the most abstract mathematics is grasping at notions of relationships and structure; extremely technical formulations of ideas that perhaps only a philosopher would love, except in those cases that physicists or other scientists discover them in their subject. Or, of course, a brilliant artist such as M.C. Escher, who without mathematical training intuited tilings of the hyperbolic plane in such works as Angels and Demons:
This is not to say that a mathematical idea is art because an artist could convey it visually: this is to say that an idea could be apprehended independently by mathematicians and visual artists, and presented in different forms, where the medium of the visual artist is the one better suited for a popular audience.
Yes, mathematics is an art, and every calculation in math class is analogous to a study of how to express or exercise some idea. If it is a little recondite, can we not say the same for contemporary art, or atonal music? If the subject or the expression is a little recherche, perhaps this puts mathematics in good company (or perhaps one might say in practise that it puts the other pieces of difficult to appreciate art in good company). But undoubtedly mathematics has many of the features of an art whose purpose is to explore an idea. Back of the envelope calculations are hasty sketches; we even call them "proof sketches", though the sketch is not of the proof but of the idea that the proof itself is meant to convey, just as a sketch is developed upon in portraiture.
If the art objects of mathematics are not as unique, it is merely because the techniques of mathematics are easy enough to grasp, for enough others, that it is an art uniquely amenable to forgery — except that we don't think in terms of forgery, because as an art the priority of mathematics is not so much unique expression as elegant expression (can you forge the art of a dancer?).
This is in part because we use the representations we create to try to better grasp at a difficult subject; and as far as pride and professionalism are concerned, first expression is what most people obsess over (as with other works of art as distinct from their forgeries).
As for the final question — can there be a melancholic mathematics, a joyful algebra? — though economics may be called a "dismal science" (where the latter part of the term is perhaps an unsupported allegation), I don't think this is the point. Does Excher's Angels and Demons have an emotional impact? Perhaps if one is strongly religious; but what of his other works, such as Metamorphosis II?
What emotion does this provoke? From personal experience, and that of many I've known, there are perhaps only two clear emotions that it provokes, which may or may not be distinct: intrigue and wonder. The audience may be narrow in which mathematics provokes such responses, but this is exactly why many people are drawn to it: for whatever reason, many are transfixed by the ability of mathematics to express relationships. While there may not be many artists such as Escher who provoke such an immediate but relatively abtracted intellectual appreciation (he's no Van Gogh or Michelangelo, after all: one mounts an Escher on the wall for different surface reasons than one does Starry Night), at least in Escher we can see that there is precedent in the visual arts, by a non-mathematician, of someone whose work provokes a very similar response — and anecdotally, in a very similar collection of people — the same general response as mathematics.
Thus I would say that mathematics is an art; and that if you insist that it is not an art, that there is then no sharp dividing line between art and mathematics, as one can gradually and subtly shift between the two subjects much as Escher shifted between the forms in his own art.
