NanoNet: Parameter-Efficient Learning with Label-Scarce Supervision for Lightweight Text Mining Model

The lightweight semi-supervised learning (LSL) strategy provides an effective approach of conserving labeled samples and minimizing model inference costs. Prior research has effectively applied knowledge transfer learning and co-training regularization from large to small models in LSL. However, such training strategies are computationally intensive and prone to local optima, thereby increasing the difficulty of finding the optimal solution. This has prompted us to investigate the feasibility of integrating three low-cost scenarios for text mining tasks: limited labeled supervision, lightweight fine-tuning, and rapid-inference small models. We propose NanoNet, a novel framework for lightweight text mining that implements parameter-efficient learning with limited supervision. It employs online knowledge distillation to generate multiple small models and enhances their performance through mutual learning regularization. The entire process leverages parameter-efficient learning, reducing training costs and minimizing supervision requirements, ultimately yielding a lightweight model for downstream inference.

💡 Research Summary

NanoNet addresses the pressing need for lightweight text‑mining models that can be trained under extreme label scarcity while keeping training costs and inference overhead minimal. The framework builds on a large‑scale pretrained encoder‑only language model (MBER‑T) as a teacher and generates multiple ultra‑compact student networks via online knowledge distillation. Unlike prior work that relies on offline distillation or full‑parameter fine‑tuning, NanoNet updates only the bias terms of the students (the BitFit strategy), thereby reducing the number of trainable parameters to less than 0.01 % of the original model.

The training pipeline consists of three tightly coupled components: (1) Online Distillation – the teacher’s intermediate representations are projected onto two‑layer BERT‑style students, producing a strong initialization with far fewer parameters; (2) Semi‑Supervised Consistency – an EMA‑based teacher provides pseudo‑labels for unlabeled data, and a standard cross‑entropy loss is applied on the tiny labeled set; (3) Mutual Learning – a cohort of K students (K ≥ 2) exchange predictions on augmented inputs, enforcing a KL‑ or MSE‑based consistency loss among peers. This peer‑to‑peer regularization raises the ensemble’s posterior entropy, which empirically improves generalization under severe supervision constraints.

Experiments were conducted on several benchmark text‑classification tasks (AG News, IMDb, SST‑2, and additional GLUE‑style datasets) with label budgets of 10, 30, 40, and 50 examples per class. NanoNet’s two‑layer student consistently outperformed the state‑of‑the‑art lightweight semi‑supervised methods DISCO and PSNET, achieving 1.2–2.5 % higher accuracy while using at least 0.9 × 10³ trainable parameters—orders of magnitude fewer than the full 12‑layer BERT or 24‑layer MBER‑T backbones. Inference latency was comparable to, or slightly better than, the baselines (≈ 1.1–1.3×). Center Kernel Alignment (CKA) analyses showed that mutual learning substantially increased representation similarity between students and the teacher, confirming that peer regularization helps the compact models inherit high‑level features from the large teacher.

Key contributions are: (i) a unified framework that simultaneously satisfies label‑scarce input, parameter‑efficient fine‑tuning, and lightweight inference; (ii) the novel combination of online distillation, bias‑only BitFit updates, and reciprocal‑instruction ensembles; (iii) extensive empirical validation demonstrating superior performance with dramatically reduced trainable parameters and computational cost.

Limitations include reliance on MBER‑T as the sole teacher architecture and potential expressivity constraints of bias‑only updates for extremely shallow networks. Future work will explore multi‑teacher setups with diverse architectures (e.g., LLaMA, DeBERTa) and integrate adapter‑style parameter‑efficient modules to further boost flexibility and performance. Overall, NanoNet offers a practical, cost‑effective solution for deploying high‑quality text‑mining models in resource‑constrained environments where labeled data are scarce.