Hidden discreteness may yield obvious discreteness
I would describe most of your examples as demonstrating discreteness at a higher level, arising from discreteness at a lower level.
The Riemann zeta function is a continuous function which encodes the properties of the primes — but it is defined as an analytic continuation of an infinite series summing over the integers; the relationship between the Riemann Zeta and the primes may be seen as arising due to the fact that for real input values for which the series is convergent, the Riemann Zeta describes something of a probability distribution over the integers in terms of the probability that it is divisible by any particular prime power, via the Fundamental Theorem of Arithmetic.
String theory, a proposed theory of particles, considers continuous
objects — which have a finite length, such as closed loops, which define a length scale and determines the possible vibrational modes.
Through QM discreetness of energy levels etc. emerge from a
continuous wave equation — in particular because the electrons are attracted by protons; where in particular all electrons have the same charge, all protons have the same charge. We would still have discrete energy levels for any one particular atom if this were not the case, but it would be much harder for us to discover that it were so, because we discovered it through bulk properties of matter (e.g. experiments with heated gasses). Still more remarkably, protons and electrons both have the same magnitude of charge.
In each case, the discreteness which we find to be "emerging", may only be emerging because of discreteness which is present at another lower level. Simple continuous mathematical objects, such as the real number line or an exponential curve, don't betray any particular sign of discreteness at all; although our only tools to describe the concept of the real line is through discrete symbols — and by demonstrating its failure to conform to any naive model of discreteness.
Discreteness and continuousness as heuristic concepts
Perhaps your remark
any theory of everything which humans can find, or any theorem of mathematics, must be formulated using a finite set of discrete symbols
is the more interesting: is it the case that any theory that we are capable of formulating will involve discreteness, and if so is it because of our notational systems? I think that mathematics — both as an example of language, and as an extension of our interest in quantity and structure — involves discreteness at its deepest levels, possibly because discretization is fruitful for solving problems quickly and well enough in the several million years in which our ancestors lived.
But it is not clear that this means that we are bound to perceive discreteness where there is none. Indeed, it took us a long time to perceive discreteness where evidently it did exist, in the form of the atomic theory of matter; and in some branches of mathematics it is much easier to approximate a large discrete object (such as an infinite series) by a continuous one (in this case, an integral), than to evaluate the discrete object directly. Here we see that sometimes it is more fruitful a strategy to compute things as though they were a continuum.
Speech recognition is an interesting example: speech elements are discrete phonemes, which are a coarse-grained description of a time-evolving frequency spectrum representing pressure waves in air, which is actually composed of individual atoms, but which themselves are peaks in the continuous wave-function, and so forth. Our best descriptions of phenomena equivocate between whether it is better to model that phenomenon as a continuum or as discrete. Furthermore the choices are essentially made by us as a matter of computational (and conceptual) convenience.
In view of the lesson of quantum mechanics, we might ask whether there is an excluded middle of something which is in some way neither discrete nor a continuum, but has features of both. Perhaps such a notion would be more informative of the way that the world works and explains why we have a tendency to alternate between continua and quanta for describing the world. But more to the point, the very notions of discreteness and continuum are tools which we use to understand the world, and our notions of what the signs of 'continuousness' and 'discreteness' are suggest to us ways in which to try to understand the world, which will not necessarily provide us insight into its fundamental nature unless we have been lucky enough that those practical concepts relate to fundamental features of the world.