Semantic Specialisation of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints

We present Attract-Repel, an algorithm for improving the semantic quality of word vectors by injecting constraints extracted from lexical resources. Attract-Repel facilitates the use of constraints from mono- and cross-lingual resources, yielding semantically specialised cross-lingual vector spaces. Our evaluation shows that the method can make use of existing cross-lingual lexicons to construct high-quality vector spaces for a plethora of different languages, facilitating semantic transfer from high- to lower-resource ones. The effectiveness of our approach is demonstrated with state-of-the-art results on semantic similarity datasets in six languages. We next show that Attract-Repel-specialised vectors boost performance in the downstream task of dialogue state tracking (DST) across multiple languages. Finally, we show that cross-lingual vector spaces produced by our algorithm facilitate the training of multilingual DST models, which brings further performance improvements.

💡 Research Summary

The paper introduces Attract‑Repel, a post‑processing algorithm that refines pre‑trained distributional word embeddings by injecting lexical constraints derived from monolingual and cross‑lingual resources. The method takes a vocabulary V together with two sets of constraints: a synonym set S (e.g., “intelligent‑brilliant”) and an antonym set A (e.g., “vacant‑occupied”). Training proceeds in mini‑batches. For each synonym pair (xₗ, xᵣ) the algorithm selects a negative example pair (tₗ, tᵣ) from the remaining vectors in the batch that are most similar to xₗ and xᵣ respectively; for each antonym pair it selects the most dissimilar vectors as negatives. The loss consists of three terms: (1) a hinge loss that forces synonym pairs to be at least δ_syn closer to each other than to their negatives, (2) a hinge loss that forces antonym pairs to be at least δ_ant farther apart than to their negatives, and (3) an L₂ regularisation term that penalises deviation from the original distributional vectors (weighted by λ_reg). This combination simultaneously pulls together synonyms and pushes apart antonyms while preserving the useful distributional information already encoded in the vectors.

A key contribution is the seamless integration of cross‑lingual constraints extracted from BabelNet. By aligning English words with their translations in other languages, the algorithm can tie multiple monolingual spaces into a single multilingual embedding space. High‑resource languages (especially English) provide abundant synonym/antonym pairs, which are transferred to low‑resource languages such as Hebrew and Croatian, dramatically improving their intrinsic quality.

The authors evaluate the approach on two fronts. Intrinsically, they test on SimLex‑999 and SimVerb‑3500, obtaining state‑of‑the‑art correlations and outperforming prior retrofitting and counter‑fitting methods. Cross‑lingual specialization yields especially large gains for the low‑resource languages, confirming the effectiveness of semantic transfer. Extrinsically, they apply the specialised vectors to Dialogue State Tracking (DST), a downstream task that requires fine‑grained understanding of user goals. Using a strong neural DST model, they show that Attract‑Repel vectors improve joint‑goal accuracy on English dialogues. They further create Italian and German DST datasets, demonstrating that the same vectors bring even larger improvements in those languages. Finally, they train a single multilingual DST model on English, Italian, and German jointly; this model outperforms each monolingual counterpart, illustrating that a unified multilingual embedding space can serve as a common semantic foundation for multi‑language dialogue systems.

The paper also discusses limitations. Languages lacking any lexical resources cannot benefit directly, although BabelNet’s wide coverage mitigates this for many languages. The selection of negative examples depends on batch size, and overly aggressive antonym constraints can over‑separate the space, hurting tasks that rely on graded similarity. The authors suggest future work on dynamic batch sampling, adaptive weighting of constraints, and zero‑shot transfer to truly resource‑scarce languages.

In summary, Attract‑Repel offers a lightweight, effective way to inject both synonymy and antonymy knowledge into existing embeddings, works equally well for monolingual and cross‑lingual settings, and yields measurable improvements on both intrinsic similarity benchmarks and a practical dialogue understanding task. Its ability to create high‑quality multilingual vector spaces opens avenues for low‑resource language support and multilingual NLP systems.