As someone with a mathematical background, you might appreciate that there's a difficulty in expressing the concept no two things are identical. In the way in which we use the word 'two' here, it's tautological — we mean to imply objects which are distinct from one another, in which case they are ipso facto not identical.
When we talk about having objects which are interchangeable, we implicitly invoke some notion of equivalence, which stands in for identity when we're prepared to ignore differences. A tarnished coin is not as valuable to a collector as one in mint condition, but so long as it's still identifiably a coin of the same denomination, they can be put to the same purpose. (Of course it would be difficult in practise to describe the precise physical qualities in which usable coins are equivalent: there is a case of the Sorites Paradox going on here, in which any division between usable and unusable coins is likely to be quite arbitrary.)
Schipol has already made reference to the identity of indiscernables, which is a criterion which describes "identity" as the most refined equivalence relation — that 'two' objects are not in fact the same object if and only if they differ somehow. This may be considered impractical in a naturalist account of metaphysics for a few reasons, though: when would you be presented with two things without being sure that they're distinct? Perhaps you're presented them at different times — but then how can you be sure that, if what you're being presented with is one object and then a similar object, that it is not the same object but changed slightly with time (because perhaps something was done to it)? Or if the object is extremely similar, how can you tell that there isn't some subtle difference which you didn't think to test?
Perhaps you would like to say that each object has its own distinct identity which sets it apart from others, however it changes and however similar other objects are to it. This, again, is the notion of haeccity mentioned by Paul Ross. But it's not clear what this gets you — it's impossible to observe the thisness of an object for comparison at a later time (it is by definition not observable as a quality), and it is undermined by such problems as Theseus' ship, which note that our notions of identity of physical objects (including our own bodies!) can not be reduced to continuity of the matter in the object — it has more to do with form, and relationships between the parts, than substance.
Is identity made exclusively up of relationships between parts? Then any two Macintosh computers are the same. You might complain that there may be incidental but discernable differences between some of the parts, but then the alternative is that changing a single screw, or making a single scratch on the case, changes the identity of the object.
One possible reaction to this is to reject the notion of identity entirely, as in Zen Buddhism, which teaches that even a notion such as "oneself", but also categorizations of objects into classes such as chairs, tables, rivers, etc. is a misunderstsnding. Perhaps it is useful to accomplish certain tasks (without remarking on what tasks are when trying to accomplish), but at the least it masks or understates the facts of the matter. Such categorizations might be best understood as a form of language game: a way to operate, and co-operate, in the world which however does not necessarily bear any truth-content, except inasmuch as the game models well features of reality. But there is no a priori reason why any particular way of speaking about the world should be expected to do so, and even some particularly effective way of speaking about the world may still have defects. Perhaps all ways of speaking about the world has deficiencies, however slight; this is the position of Daoism, for instance. And however useful for everyday discourse, perhaps thre notion of identity is one of these defects. But then why is the notion of identity a useful one — is there not a feature of the world which it identifies, or is there not a feature which it at least approximates? It's unclear.
In summary, the question of identity (or its negation) is either trivial, problematic, or complete nonsense, depending on what you demand from it.