How distant a culture from ours would you intend to target?
What is possible - for about any human culture - is to create a pictorial dictionary where nouns, verbs and adjectives are defined through images or set of images depicting them. After establishing the vocabulary, pictorial stories could define grammar giving example sentences. Once initial set of language is established, it could be used for self-expansion.
This still depends on the targets being able to recognize the actions and objects depicted, and even on them possessing a sense of sight similar to ours.
Yes, pure mathematics is very difficult to describe complex but down-to-Earth concepts - show a picture of a person with one leg extended, the other trailing, off balance and you have the definition of "walk". But scientists spent decades trying to program functional walking robots - in other words, define "walking" in terms of mathematics. And what if your target audience is only ever rolling or floating in water? How would you depict "walking" for them in a way they'd understand?
So, while creating a language deriving its complete meaning from mathematics would be extremely, impossibly difficult, it's still quite possible to create a self-describing language based on mathematics, that contains such trivial concepts as "nitrogen", "wavelength spectrum", "light year", "electromagnetism", "fission", "atmosphere", "Oberth maneuver", "Planck time", while being utterly helpless with complex ideas like "sneeze" or "cookie".
Edit: since it's far too long for a comment...
First, you establish the language of mathematics: notation of numbers, operators, the whole "grammar" of mathematic expressions. We've established this is perfectly doable.
Then draw a hydrogen atom, dimensions described in multiples of Planck Length. Any civilization understanding proportions, knowing composition of hydrogen and knowing Planck Length will easily recognize the familiar dependency, both recognizing the unit of distance and the type of atom, together with symbols for proton and electron.
Add deuterium, with its mass, next to hydrogen's mass; we have a unit of mass and a symbol for neutron.
Depict emission of a photon to obtain its symbol, a unit of energy, a unit of frequency and a unit of time.
Armed with geometry, electron, proton and neutron you can define chemistry, all of the universe's composition. Using chemistry and geometry you can begin depicting physical macroscopic objects.
Armed with time and distance you can describe motion and change. Together with motion comes temperature. Wavelengths define colors. Photon layout of reflected light creates image/sight.
With motion, mass and location you can define gravity.
Define air (mix of oxygen and nitrogen, temperature, inter-particle distances) and derive pressure. Add waves to define sound.
With composition, temperature and mass you can describe stars. Frequency of pulsars will give a firm macroscopic coordinate system.
After describing our Sun, you can get to describing Earth. Sphere, air, water, soil, hot core, surface temperature and pressure.
Afterwards, proteins and DNA for describing life. Plants (chlorophyll, unmovable), animals (hemoglobin, moving). Human.
Afterwards, depicting how human's body works (eyes as light sensors, ears as sound sensors, hands and legs as actuators) you can proceed to technology. Physics and chemistry will be essential.
And so on, and so on - explaining how each element fits with the ones explained so far, you can build the whole dictionary. And yes, while possessing descriptions of motion and gravity, concepts of "Oberth Maneuver" are easy to explain. But to understand "cookie" you must know plants, grasses, grains, seeds, grinding, flour, sugar, baking, human digestion, sense of taste, and concept of pleasure. And while some are simple enough (taste = chemical analysis), others like pleasure will be incredibly hard and lengthy to describe, delving into evolution of instinct, separation of conscious mind from primal desires, and a whole course in human psychology.