Comparing human and automatic thesaurus mapping approaches in the agricultural domain

Knowledge organization systems (KOS), like thesauri and other controlled vocabularies, are used to provide subject access to information systems across the web. Due to the heterogeneity of these systems, mapping between vocabularies becomes crucial for retrieving relevant information. However, mapping thesauri is a laborious task, and thus big efforts are being made to automate the mapping process. This paper examines two mapping approaches involving the agricultural thesaurus AGROVOC, one machine-created and one human created. We are addressing the basic question “What are the pros and cons of human and automatic mapping and how can they complement each other?” By pointing out the difficulties in specific cases or groups of cases and grouping the sample into simple and difficult types of mappings, we show the limitations of current automatic methods and come up with some basic recommendations on what approach to use when.

💡 Research Summary

The paper investigates the relative merits and drawbacks of human‑generated versus automatically generated thesaurus mappings in the agricultural domain, using the multilingual AGROVOC vocabulary as a focal point. Three major Knowledge Organization Systems (KOS) are considered: AGROVOC (FAO), the National Agricultural Library Thesaurus (NALT), and the German Schlagwortnormdatei (SWD). Two distinct mapping projects are compared.

The automatic approach draws on the Ontology Alignment Evaluation Initiative (OAEI) 2007 “food task”, in which five systems—Falcon‑AO, RiMOM, X‑SOM, DSSim, and SCARLET—produced mappings between AGROVOC and NALT. All systems were limited to the SKOS mapping relations exactMatch, broadMatch, and narrowMatch, which correspond to thesaurus constructs USE, BT, and NT. Performance metrics (precision and recall) show that Falcon‑AO achieved the highest precision (0.84) and a moderate recall (0.49) but could only discover equivalence relations; it missed hierarchical links and any mappings requiring background knowledge. RiMOM and X‑SOM performed considerably worse (precision 0.62/0.45, recall 0.42/0.06). DSSim showed modest results (precision 0.49, recall 0.20). SCARLET was the only system to generate hierarchical matches using the Watson semantic‑web search engine, yet its mappings were overly generic, leading to a near‑zero recall (0.00). Overall, the automatic methods excel at simple synonym (1:1) matches but struggle with broader/narrower relations, multilingual nuances, and domain‑specific background knowledge.

The human approach originates from the GESIS‑IZ “KoMoHe” project, which produced a fully bilateral mapping between AGROVOC and SWD. Mapping types include equivalence (=), broader (<), narrower (>), associative (^), and null (0), mirroring SKOS’s exactMatch, broadMatch, and narrowMatch. The process involved extensive use of intra‑thesaurus relations, scope notes, synonym dictionaries, and expert domain knowledge. In total, 6,254 AGROVOC start terms were linked to 5,500 SWD end terms, yielding 4,557 exact matches, 100 broader, 314 narrower, 337 associative, and 3 null relations. The mapping was reviewed by terminology experts and empirically tested for information‑retrieval recall and precision, confirming high quality across all categories.

A qualitative assessment aligns the overlapping AGROVOC terms that appear in both the automatic (AGROVOC‑NALT) and manual (AGROVOC‑SWD) mappings, grouping them into four thematic clusters: Taxonomic, Biological/Chemical, Geographic, and Miscellaneous. Approximately 5,000 AGROVOC terms are common to both projects. The analysis reveals that automatic systems achieve 70‑80 % success on simple synonym mappings, but their performance drops below 10 % for complex hierarchical or associative mappings. Human experts maintain >95 % accuracy across all categories, especially excelling in cases requiring background knowledge, cross‑language lexicalization (e.g., “Oryza sativa” vs. Chinese “稻作”), or nuanced domain distinctions.

From these findings the authors draw several practical recommendations:

1. Hybrid workflow – employ automatic tools to generate a first‑pass set of equivalence mappings, then have human experts validate, refine, and extend them, particularly for hierarchical, associative, or multilingual relations.
2. Domain‑specific prioritization – allocate human resources to areas where automatic recall is low (Biological/Chemical, Geographic) and rely on automation for high‑recall domains (Taxonomic).
3. Tool improvement – enhance automatic systems with background‑knowledge bases, richer ontological links, and robust multilingual alignment techniques to reduce the gap in complex cases.
4. Cost‑benefit balance – recognize that while manual mapping delivers superior quality, its scalability is limited; a combined approach maximizes coverage while controlling labor costs.

In conclusion, the study demonstrates that automatic thesaurus mapping can efficiently handle large‑scale, straightforward synonym alignment, but human expertise remains indispensable for ensuring semantic precision in hierarchical, associative, and multilingual contexts. By integrating both approaches, institutions can achieve high‑quality, scalable vocabulary interoperability in the agricultural information domain.