A prerequisite for your question is the question "is what the neural networks do really 'thinking?'" Obviously this question is contentious even still (although each year more and more researchers seem to think "yes"), with Searle's Chinese room argument being the canonical objection. It seems that if the status of AI, specifically neural networks, as something that 'thinks' is rejected then we cannot use Google's recent findings to support the language of thought hypothesis, because we don't have a model of something using thought. If, however, we do agree that a neural network, or at least a sufficiently powerful more complete AI system that uses neural networks, does "think," then this might be a showing of evidence for a language of thought. It seems that if we reject that AI can think we are stuck with a no answer, but if we accept that they can think there is room for discussion as to whether this does or does not support the idea. Considering the dead-ended conversation if we reject AI as being able to think, we should proceed by accepting it as a correct model of thought (whether it is synonymous with human thought or not can be left aside).
UMD professor Christoph Schulze defines the language of thought hypothesis as follows:
The language of thought hypothesis is part of a larger canon of theses(See [2] for the full list) and can be
summarized as.
Mental representations are structured
Parts of these structures are transferable, that means they can appear in different representations
Mental representations have a compositional semantic. Which means that the meaning of complex
representations can be extracted out of its parts.
The language of thought, which is sometimes called “Mentalese” is thought to have a structure much like
regular languages.
The SEP and IEP have similar definitions:
LOTH is an empirical thesis about the nature of thought and thinking. According to LOTH, thought and thinking are done in a mental language, i.e., in a symbolic system physically realized in the brain of the relevant organisms.
The language of thought hypothesis (LOTH) is the hypothesis that mental representation has a linguistic structure, or in other words, that thought takes place within a mental language. The hypothesis is sometimes expressed as the claim that thoughts are sentences in the head.
As mentioned in the question, the paper Google researchers released states:
5.1 Evidence for an Interlingua:
Several trained networks indeed show strong visual evidence of a shared representation. For example, Figure 2 below was produced from a many-to-many model trained on English↔Japanese and English↔Korean. To visualize the model in action we began with a small corpus of 74 triples of semantically identical cross-language phrases. That is, each triple contained phrases in English, Japanese, and Korean with the same underlying meaning.[...] Inspection of these clusters shows that each strand represents a single sentence, and clusters of strands generally represent a set of translations of the same underlying sentence, but with different source and target languages.
This directly supports the definitions given above for a language of thought. The systematic way that the programs cluster the languages together by the internal semantics of the statements, if completely true, can be thought of as a physically realize, linguistic system contained in the structure of the neural networks. However, the paper goes on to state results that this semantic clumping was not perfect:
5.2 Partially Separated Representations
Not all models show such clean semantic clustering. Sometimes we observed joint embeddings in some regions
of space coexisting with separate large clusters which contained many attention vectors from just one language
pair.
For example, Figure 3a shows a t-SNE projection of attention vectors from a model that was trained
on Portuguese→English (blue) and English→Spanish (yellow) and performing zero-shot translation from
Portuguese→Spanish (red). This projection shows 153 semantically identical triples translated as described
above, yielding 459 total translations. The large red region on the left primarily contains zero-shot
Portuguese→Spanish translations. In other words, for a significant number of sentences, the zero-shot
translation has a different embedding than the two trained translation directions. On the other hand, some
zero-shot translation vectors do seem to fall near the embeddings found in other languages, as on the large
region on the right.
It is natural to ask whether the large cluster of “separated” zero-shot translations has any significance. A
definitive answer requires further investigation, but in this case zero-shot translations in the separated area
do tend to have lower BLEU scores.
The uncertainty of the results is a clear indicator that this is not, of course, an irrefutable proof of the language of thought hypothesis. The hypothesis states that all mental thoughts have a linguistic structure which is contained purely in the mental facility of the thinker. We won't know if the neural networks truly do have this interlingua until we have more data from future experiments.
A suggestion for further experiments would be to test the hypothesis presented by Schulze that:
Mental representations have a compositional semantic. Which means that the meaning of complex representations can be extracted out of its parts.
The neural networks proved to be able to cluster together semantically similar sentences with good probability, but proving their ability to separate the individual semantic components of the sentences into well defined clusters would be much more powerful.
One other important question to ask is the question "do these results hold any implication for the objections to the language of thought hypothesis?" If these results could somehow answer some of the objections they would be much stronger support for the hypothesis. The IEP defines a potentially fatal problem for the language of thought hypothesis:
a. Individuating Symbols
An important and difficult problem concerning LOTH is the individuation of primitive symbols within the language of thought, the atomic mental representations. There are three possibilities for doing so: in terms of the meaning of a symbol, in terms of the syntax of a symbol (where syntactic kinds are conceived of as brain-state kinds), and in terms of the computational role of the symbol (for example, the causal relations the symbol bears to other symbols and to behavior).
This objection seems to hit on the question I posed above for future research. If the neural networks are unable to parse individual semantic atoms then we would not be able to reliably say they are using a language of thought. Again, this needs a lot more testing.
In summation the neural network results do provide a lot of similarities with the language of thought hypothesis and it is reasonable to assume this as a stepping stone towards some greater support for the hypothesis. More work should definitely be done in this area (even if it turns out to only have the practical importance of creating better language translation programs).
