Multilingual Sentence Representations

This post is an experiment killed rather early while I was looking into interpretability of learned transformer representations during my Masters. The post is added to my personal blog on 2020 August 06, but the date of the original post in my blog preserved. It’s written in the fashion of somthing like a paper. A better work I found after dropping my pursuits surrounding the problem is Investigating Multilingual NMT Representations at Scale, by a group in Google.

Introduction

In this work, we investigate

• Sentence representations from a multilingual neural machine translation model.
• We explore the possibilities of re-use and transfer of these learnt embeddings.

Methodology

Datasets and Training

We train a multilingual model with joint encoder, decoder and shared embeddings between both as per Johnson et al. (2017) on a large compiled corpus consisting of IIT-Bombay Hindi English Parallel Corpus (IITB-hi-en) [Kunchukuttan et al. (2018)], Indian Languages Corpora Initiative (ILCI) [Jha (2010)], WAT-Indic Multi Parallel Corpus (WAT-ILMPC) [Nakazawa et al. (2018)]. The test sets of both IITB-hi-en and WAT-ILMPC are left out of training and the evaluation scores on these are reported below. We use a compilation of validation sets from the two datasets for monitoring validation loss.

For evaluating multilingual embeddings we use the Mann Ki Baat dataset as an unseen dataset, and ILCI corpus from the training dataset. The details of this dataset are described in the table below. We use 3348 samples aligned across each language form Mann Ki Baat and 50K from ILCI corpus. The BLEU scores on Mann Ki Baat are provided below.

Encoded Representations

To obtain fixed-dimensional sentence representations for a variable length input sequence, we follow Zhang et al. (2018) to use a concatenation of max-pooled features over time and the mean across the time-steps each individually L2 normalized. Below, we detail an ablation study by varying possibilities and found that this representation gave best results. Once we reaffirm the extracted features work best among those proposed, we attempt to visualize the Mann Ki Baat dataset in lower dimensions to seek what properties are strong in the encoded representations at a macro scale.

Nearest Neighbours

First, we attempt to analyze over the test set for a give source and target in languages \(xx\), \(yy\) respectively, embeddings generated to translation to which languages \(\hat{xx}\), \(\hat{yy}\) respectively works the best. To this end, we collect the embeddings from encoder representations for each source attempted to translate to a language. For each sample, we obtain the nearest neighbours in other languages using faiss¹ (Johnson et al., 2019).

For the above experiment, we report the overall precision@1 over each pair of languages in the table below. The columns represent the source whose language of the sample whose nearest neighbour is being retrieved. The row indicates the target language samples which are being queried. Precision is computed in the information retrieval sense as

\[ \mathrm{p@k} = \frac{\#(\mathrm{relevant} \cap \mathrm{retrieved})}{\# \mathrm{retrieved}} \]

A retrieval is considered relevent if it belongs to the same multilingual sample as the query.

Why the above features are best? Expand P@1 tables for other features.

(But the above is among very little samples. Perhaps I should index with larger representations, to well establish the Mann Ki Baat ones still come close.)

Embedding and visualizing in lower-dimensions

In the below images, we illustrate the lower-dimension projections using t-SNE of the sentence embeddings from Mann Ki Baat dataset. We first do a PCA to reduce the dimensions to 20, followed by t-SNE on 2-dimensions.

The representations are built from multilingual samples of Mann Ki Baat, each entry having attributes identified by the unique tuple source, target and sample-id. Multiple language samples of the same content are indexed by same sample-id.To observe which axis among these forms the grouping which make most sense - we fix a few, while keeing the others varying.

Note that our methods of embedding to a 2D space of all samples are unsupervised. We proceed to color the resulting datapoints according to the labels extracted as source, target, sample-id and combinations of the same.

Target Space

In the first case, we color the projections using the target language attribute.

The best grouping is obtained on the target space, implying each the encoder representations group same target language samples together. The results are in agreement with the findings of Johnson et al. (2017). We’ve possibly learnt the hard way that these sentence representations aren’t that reusable for vector space comparisons as cross lingual retrieval mechanisms. Another interesting thing to note here is that each representations contain the same shapes for all languages.

(TODO: This could suggest that there exists a matrix which can be obtained by solving the Procustes problem similar to Lample et al. (2018) to get a rotation matrix. Update: Google’s submission did SVCCA, which might be more apt a choice.)

Source language space

The source language boundaries still form some subclusters, although not as the target. For a better look, we try to view the source samples embedded for a fixed target language. Given below are encoded representations while attempting to translate to English and Hindi as indicated in the graphs below.

It’s observable above that en-t2en directly maps to hi-t2hi and en-t2hi somehow is similar to hi-t2en. But the samples not being overlayed on top of each other is still a problem, as attempts to translate to a single language isn’t likely to produce robust embeddings for cross-lingual sentence representations.

Colored by source and target

The above is a more fine-grained view of the clusters, considering src-tgt as the labels and coloring accordingly.

Same concepts - different language.

Next, we encode all sentences to translate to Hindi (xx-t2hi). Since here points are closer in terms of source language rather than similarity in meaning or content, we can conclude that even at a microscopic scale, we’ll need further refinements to use these embeddings as multilingual sentence representations.

Above, we check if multilingual closeness exists, and turns out it doesn’t. English and Hindi seems to have their own close spaces, while the others are a bit cluttered. This also seems to correlate with the size of data we have in each languages, the degree of intra-language closeness and inter-language separation. Perhaps once the data matches up in other languages, we could see how this evolves.

Literature Survey

Multilingual Translation

Johnson et al. (2017) introduced Multilingual Neural Machine Translation switching target language based on a token prepended to the input sequence. This simple method demonstrated major improvements and zero-shot capability in translation. If the parameters of the encoder and decoder and respective embeddings are shared among all languages, a consequence is that the encoder outputs become cross-lingual representations of the concept in the source language.

Feature Extraction

Zhang et al. (2018) adapts the encoder architecture from the transformer model proposed by Vaswani et al. (2017) as sentence representation learning models. They however investigate the utility through downstream tasks constructing the sentence-representations through an autoencoding objective on the sentences. Schwenk and Douze (2017) uses max-pooled features across timesteps, which seems to be working out well for their use cases.

Mining Parallel Pairs

Past works in multilingual mining have made significant use of representations arising out of translation task [Schwenk and Douze (2017), Schwenk (2018), Artetxe and Schwenk (2019a), Artetxe and Schwenk (2019b)]. Since mining is enabled by the ability to query a sample in the vector space induced by translation, most of these work becomes relevant to the likes of cross-lingual sentence embeddings.

Schwenk and Douze (2017) proposes the following desired properties: (i) multilingual closeness, (ii) semantic closeness, (iii) preservation of content, (iv) scalability to many languages. Schwenk (2018) uses cosine similarity, for the same representation and applies it to the BUCC task. They conclude the distance can be used in confidence estimation or to filter backtranslations. Artetxe and Schwenk (2019a) strips the encoder inputs off source or target language information, having embeddings corresponding to target fed to the decoder instead, thereby providing “encoded-representations” of the sequence in a common space.

• The error rates reported are low. Quite unsure if our embeddings on Mann Ki Baat matches up to this degree. The nearest neighbour retrieval precision presented before as proxy supports this. Uses vanilla seq2seq NMT models (possibly Bahdanau et al. (2014) or Luong et al. (2015)).
• It may perhaps be interesting to look at the t-SNE dynamics of this modified network.

Evaluations

The XNLI Dataset [Conneau et al. (2018)] seems to be used by people in the community for benchmarking multiingual embeddings [Artetxe and Schwenk (2019b)]. Hindi (hi) seems to be the only reported Indian Language in the testing set.

Often, mining parallel text from news corpus and training a translation system to obtain better BLEU seems to be an assertion in favour of better mining methods. If the representations are robust and performing well, this could indicate their success.

Conclusion and Future Work

Artetxe and Schwenk (2019b) seems to be a good way to go to generate sentence embeddings, but requires training from scratch and some modifications to pytorch/fairseq.

References