There are various theories about the origin of languages. Personally I do not believe in things like Monogenesis, but rather think that languages start automatically with communicating basic evolutionary needs and then evolve eventually in abstraction and finetuning.
A good basis to study the origin of languages are animals and their various ways to communicate with their children, mates, friends, rivals etc. and in fact scientists spend years of their life to listen to lets say birds, like pink flamingos.
Another interesting study is to look at human languages and their finetuning. In particular the various forms of mathematics in different populations and their result on mathematical understanding is ethnomathematics. For example the Maya had two kinds of zero’s, namely – loosely speaking – a zero for the begin of a count (in particular they counted days) and a zero for denoting that a timespan doesn’t count – an article on this can be found in this issue of Spektrum der Wissenschaft.
Somewhat interesting is the reverse thing i.e. to speak of language in mathematical terms. E.g. there exist a mathematical structure called “category” (which is defined in the Wikipedia link). Let us assume we have
-a collection of objects, namely sets of words (call them for simplicity languages).
-in addition for any pair language1 and language2 one has a bunch of translators (in mathematical terms a set of morphisms) who translate e.g. language1 into language2
Let us depict this translation with an arrow in the following way:
language1 —translator—> language2
the language and the translators come together with a “repeater” (mathematically an identity morphism), i.e. someone who just translates the same language into the same language and a “composite” that is a translator who replaces two translators, i.e. if fj is a translator from french to japanese and je is a translator from japanese to english (we assume that the two translators can only translate in one direction) then fe would be a composite for fj and je, i.e. someone who does the same job as fj and fe one after the other, namely to translate from french to japanese.
So again one could depict this as:
japanese —je—> english
Now this example is a category if and only if the following is true:
1.)(mathematics: “left and right unit law”): first translate and then repeat is the same as first repeat and then translate is the same as just translate
2.)(mathematics: “associative law”) associativity holds.
If the red and the green arrows are the same then “associativity holds” or in other words a translation from english to french would give the same result if we first translate english into german and then translate german into french or if we first translate english into japanese and then to french. If this is true we have an example of a category. If not (as it is usually the case for languages) then not.