Probing the Natural Language Inference Task with Automated Reasoning Tools

💡 Research Summary

The paper investigates whether a logic‑based pipeline can be used to solve the Natural Language Inference (NLI) task, focusing on the Stanford Natural Language Inference (SNLI) corpus. The authors employ Attempto Controlled English (ACE), a machine‑oriented controlled natural language that can be translated directly into first‑order logic (FOL) via the Attempto Parsing Engine (APE). Their workflow is as follows: (1) each premise‑hypothesis pair from SNLI is fed to APE; if parsing fails, a set of eight syntactic rewrite rules (R1‑R8) is applied to transform the sentences into ACE‑compatible form, after which parsing is attempted again. (2) Successfully parsed sentences are converted into TPTP‑format FOL formulas (P for premise, H for hypothesis) and given to a first‑order resolution prover. The prover checks whether H or ¬H can be derived from P within a clause limit of 1500. (3) If the prover yields no decision, two groups of semantic augmentation rules are applied. The first group (S1) adds hypernym implications for nouns using WordNet, inserting formulas of the form ∀x n₁(x)→n₂(x) into an auxiliary set A, and adds negative equivalences ∀x n₁(x)↔¬n₂(x) into a set N when no hypernym relation exists. The second group (S2) does the same for verbs, creating predicate‑level implication or negation formulas. The augmented sets are then fed back to the prover, and if still no conclusion is reached the pair is labeled “neutral”.

The syntactic rules address several recurring obstacles: (R1) noun phrase adjective chaining, (R2) coreference resolution (replacing pronouns with unique proper‑noun identifiers prefixed by “p:”), (R3) conversion of past‑tense verbs to present, (R4) spelling out cardinal and ordinal numbers, (R5) removal of predeterminers, (R6) swapping adverbial phrases that precede verbs, (R7) normalizing “but”/“yet” conjunctions to “and”, and (R8) simplifying present/past continuous constructions. These transformations preserve the original meaning as much as possible while making the sentences acceptable to ACE.

Empirically, the authors evaluate on a 10 000‑pair development subset of SNLI. Without any rewrite rules, APE parses only 7.06 % of sentences; after applying the syntactic rules, coverage rises to 16.61 %. For the subset that is successfully parsed, the prover’s entailment predictions are perfectly accurate (100 % precision), but overall classification accuracy remains low at roughly 28.7 %, far below current deep‑learning baselines and even a random guess. The low accuracy is largely due to the fact that ACE‑generated formulas are predominantly positive descriptions and rarely contain negation, making contradiction detection difficult. Adding the semantic hypernym rules modestly improves contradiction prediction to 37.5 % accuracy, still only marginally better than chance. The authors also discover several mislabeled examples in SNLI, illustrating that a logic‑based approach can expose annotation errors that are invisible to statistical models.

In conclusion, the study demonstrates that a logic‑centric NLI system can dramatically increase the proportion of sentences that can be formally reasoned about through relatively simple syntactic rewrites, but the overall performance is still far from state‑of‑the‑art neural models. The authors suggest future work on refining rewrite rules, incorporating richer semantic resources (e.g., frame semantics, predicate‑argument structures), and hybridizing with natural‑logic systems such as LangPro, Monolog, or NaturalLI. They also propose using the presented pipeline as a diagnostic tool for dataset quality and as a baseline for subsequent logic‑based NLI research.