A hybrid formalism to parse Sign Languages

Sign Language (SL) linguistic is dependent on the expensive task of annotating. Some automation is already available for low-level information (eg. body part tracking) and the lexical level has shown significant progresses. The syntactic level lacks annotated corpora as well as complete and consistent models. This article presents a solution for the automatic annotation of SL syntactic elements. It exposes a formalism able to represent both constituency-based and dependency-based models. The first enable the representation the structures one may want to annotate, the second aims at fulfilling the holes of the first. A parser is presented and used to conduct two experiments on the solution. One experiment is on a real corpus, the other is on a synthetic corpus.

💡 Research Summary

The paper addresses the longstanding challenge of automatically annotating syntactic structures in sign languages (SL), which differ fundamentally from spoken languages due to their simultaneous use of multiple articulators and complex temporal‑spatial synchronizations. Existing work has largely focused on low‑level tracking (body parts, activity detection) or lexical‑level recognition, leaving the supra‑lexical (syntactic) level under‑annotated and lacking robust models. To bridge this gap, the authors propose a hybrid formalism that can represent both constituency‑based and dependency‑based grammars within a single framework, and they implement a parser that exploits this formalism for semi‑automatic annotation.

Core Formalism
The formalism is inspired by Context‑Free Grammars (CFGs) but departs from them in three essential ways:

1. Set‑based RHS – The right‑hand side of a production is a set of units rather than an ordered string, reflecting the non‑sequential nature of SL.
2. Detectable vs. Non‑detectable Units – Units may be directly observable by external detectors (detectable) or inferred (non‑detectable). Both can be atomic or composite.
3. Patterns and Alternatives – Instead of traditional non‑terminals, the grammar uses patterns (which encode constituency relations) and alternatives (which encode optional or mutually exclusive structures). Each pattern can be instantiated multiple times, and each instance receives a role name to avoid ambiguity.

Constraints are a first‑class component of the formalism. Temporal (start/end), spatial (hand position), and articulatory (e.g., bi‑manual) attributes are expressed as logical predicates that restrict the admissible values for each unit. The authors deliberately keep the constraint language independent of any specific logic, allowing the integration of multiple logics (Allen’s interval algebra for time, RCC for space, etc.) as needed.

Handling Incomplete Models
Because fully specified SL grammars are unavailable, the authors introduce edge nodes (e.g., “unmodeled‑loc”) that act as placeholders for structures not covered by the current model. These edges enable the parser to continue inference while flagging gaps that may be resolved interactively by a human annotator or by invoking external detectors on demand.

Parser Architecture
The parser builds an internal AND/OR graph (the “implicit graph”) where patterns correspond to AND nodes and alternatives to OR nodes. This graph is then expanded into an explicit graph whose nodes represent concrete instances of units. Parsing proceeds top‑down from a set of root units (e.g., a “Signing” activity detector or pre‑detected lexical signs). As the parser traverses the graph, it may request additional detections from external modules, thereby narrowing search spaces. The output is a set of solution graphs; each graph encodes a possible annotation, with attributes (time‑start, time‑end, etc.) attached to every node. When the model is under‑constrained, multiple solutions—including partial or spurious ones—are produced.

Experiments
Two experiments evaluate the approach:

1. Real‑world Constituency Test – Using excerpts from the French Sign Language (LSF) portion of the Dicta‑Sign corpus, the authors define five patterns (buoy, acknowledgment, question, sign‑check, etc.) and run the parser with manually supplied detections for the detectable units. The corpus contains only a handful of occurrences, precluding statistical significance, but qualitative analysis shows that the parser generates many solutions, some of which are false positives caused by incorrect hierarchical ordering or insufficient lower‑level modeling. Simple ranking by solution size effectively filters out many partial solutions.
2. Synthetic Dependency Test – Synthetic dependency grammars are generated, and corresponding synthetic corpora are parsed. Here the parser achieves higher recall and precision, yet still produces false positives, illustrating that even with clean dependency structures the lack of strong constraints can lead to over‑generation.

Insights and Future Directions
The study demonstrates that a unified formalism can accommodate both constituency and dependency representations, offering a flexible substrate for semi‑automatic SL annotation. However, the experiments reveal that constraint strength is crucial: weak or missing constraints yield many spurious parses. The authors suggest augmenting the grammar with recursive constraints (e.g., “a locution that forms a question cannot itself contain another question”) and exploring automatic learning of constraint parameters from larger annotated corpora. Moreover, tighter integration with real‑time detectors and a more sophisticated ranking or scoring mechanism could improve the selection of the most plausible solution among many.

In summary, the paper contributes a novel hybrid grammar formalism tailored to the multimodal, simultaneous nature of sign languages, implements a prototype parser that can operate with incomplete models, and provides initial empirical evidence of its feasibility. The work lays groundwork for future research aimed at scaling up SL syntactic annotation, refining constraint specifications, and ultimately enhancing downstream tasks such as sign recognition and translation.