To start with the easy answer: OM is not the first letter in Devanagari script, the script of Sanskrit. The first letter in Devanagari is the short vowel 'a'. OM is composed by two letters, the vowel 'au' followed by the consonant 'ma'.
One of the first records of the syllable in a written text is the beginning of Chandogya Upanishad:
OM - one should venerate the High Chant as this syllable, for one begins the High Chant with OM.
The context of this passage is a certain Vedic ritual. The Chandogya Upanishad is one of the oldest Upanishads and dates from 7./6. century BCE. It is part of the Samaveda, one of the four Vedas. Apparently, OM as a distinguished syllable is much older than this textual evidence.
The meaning of OM is unknown. Probably just on account of this, OM is the object of many speculations. The syllable is highly ranked by both Hindus and Mahayana Buddhists.
Considered from a rational point of view, OM is an example of magic speech.
Added. The Mandukya Upanishad is a prominent work which speculates exclusively about the meaning of OM. Its speculations are neither supported by etymology nor backed by philosophical arguments. The Mandukya Upanishad starts
OM - this whole world is that syllable!
The Upanishad is a late work, from about the end of the first century BCE or the beginning of the common era.