AWED-FiNER: Agents, Web applications, and Expert Detectors for Fine-grained Named Entity Recognition across 36 Languages for 6.6 Billion Speakers

Named Entity Recognition (NER) is a foundational task in Natural Language Processing (NLP) and Information Retrieval (IR), which facilitates semantic search and structured data extraction. We introduce \textbf{AWED-FiNER}, an open-source collection of agentic tool, web application, and 53 state-of-the-art expert models that provide Fine-grained Named Entity Recognition (FgNER) solutions across 36 languages spoken by more than 6.6 billion people. The agentic tool enables routing multilingual text to specialized expert models to fetch FgNER annotations within seconds. The web-based platform provides a ready-to-use FgNER annotation service for non-technical users. Moreover, the collection of language-specific extremely small open-source state-of-the-art expert models facilitates offline deployment in resource-constrained scenarios, including edge devices. AWED-FiNER covers languages spoken by over 6.6 billion people, ranging from global languages like English, Chinese, Spanish, and Hindi, to low-resource languages like Assamese, Santali, and Odia, along with a specific focus on extremely low-resource vulnerable languages such as Bodo, Manipuri, Bishnupriya, and Mizo. The resources can be accessed here: Agentic Tool (https://github.com/PrachuryyaKaushik/AWED-FiNER), Web Application (https://hf.co/spaces/prachuryyaIITG/AWED-FiNER), and 53 Expert Detector Models (https://hf.co/collections/prachuryyaIITG/awed-finer).

💡 Research Summary

The paper presents AWED‑FiNER, an open‑source suite that combines an agentic toolkit, an interactive web application, and a collection of 53 fine‑tuned expert models to deliver fine‑grained Named Entity Recognition (FgNER) for 36 languages spoken by more than 6.6 billion people. The authors identify a critical gap in current NER research: most state‑of‑the‑art systems focus on high‑resource languages, leaving the majority of the world’s speakers, especially those of low‑resource and vulnerable languages, without adequate tools. To address this, they assemble a multilingual ecosystem that supports languages ranging from English, Chinese, Spanish, and Hindi to Assamese, Santali, Odia, and extremely low‑resource languages such as Bodo, Manipuri, Bishnupriya, and Mizo.

Model Collection
The expert models are built on three pre‑trained multilingual encoders—XLM‑R‑large, MuRIL‑large, and IndicBERT v2‑MLM‑SamtLM. Each language is paired with the most suitable fine‑tuning dataset among seven publicly available FgNER corpora: MultiCoNER2, FewNERD, CLASSER, SampurNER, FiNERVINER, APTFiNER, and FiNE‑MiBBiC. Fine‑tuning is performed on an NVIDIA A100 GPU for six epochs with a batch size of 64, using AdamW (learning rate 5e‑5, weight decay 0.01). All models are deliberately kept small, each containing fewer than 355 million parameters, which enables deployment on edge devices and in offline scenarios. Performance is evaluated using macro‑averaged F1; Table 1 shows that high‑resource languages achieve macro‑F1 scores above 80 %, while low‑resource and vulnerable languages still reach the 60 %–70 % range, substantially outperforming prior single‑model baselines.

Agentic Toolkit
The toolkit is implemented with the smolagents framework. It automatically detects the language of an input text, selects the appropriate expert model from a metadata‑driven registry, and invokes it with a single API call. This “one‑line” integration is designed to fit seamlessly into larger LLM‑driven pipelines, enabling autonomous workflows that require precise entity extraction without manual model selection.

Web Application
The interactive web interface is hosted on Hugging Face Spaces and built with Gradio. Users upload or type text, choose a language, and receive real‑time visualizations of fine‑grained entity spans, color‑coded by type. The backend dynamically loads the required compact model, minimizing memory consumption. The design provides a low‑barrier entry point for non‑technical users while also serving as a benchmark platform for evaluating multilingual FgNER performance under constrained resources.

Contributions and Impact

1. Language Coverage – 36 languages covering 6.6 billion speakers, including many previously unsupported low‑resource languages.
2. Model Efficiency – All models are under 355 M parameters, facilitating deployment on CPUs, mobile devices, and edge hardware.
3. Unified Access – A single agentic API and a user‑friendly web UI give both developers and end‑users consistent access to state‑of‑the‑art FgNER.
4. Open‑Source Availability – The models, toolkit, and web app are all released on Hugging Face Hub, encouraging reproducibility and community extensions.

Limitations and Future Work
The authors acknowledge several constraints: (i) despite being “lightweight,” the models may still be too large for ultra‑low‑power microcontrollers, suggesting the need for pruning or quantization research; (ii) evaluation relies solely on macro‑F1, which masks performance on rare sub‑types and does not address label imbalance; (iii) the language‑detection‑based routing may misroute code‑mixed inputs, indicating a potential area for mixed‑language routing strategies. Future directions include exploring more aggressive model compression, incorporating hierarchical or per‑type evaluation metrics, and extending the system to downstream tasks such as relation extraction and knowledge‑graph construction.

Conclusion
AWED‑FiNER represents a significant step toward democratizing fine‑grained entity recognition across the linguistic long tail. By delivering a compact, multilingual model zoo, an autonomous routing toolkit, and an accessible web service, the authors provide a practical, scalable solution that can be integrated into modern AI pipelines and deployed in resource‑constrained environments. The work not only advances the state of multilingual NER but also sets a foundation for future research on efficient, inclusive language technologies.