Entropy-Based Data Selection for Language Models

Modern language models (LMs) increasingly require two critical resources: computational resources and data resources. Data selection techniques can effectively reduce the amount of training data required for fine-tuning LMs. However, their effectiveness is closely related to computational resources, which always require a high compute budget. Owing to the resource limitations in practical fine-tuning scenario, we systematically reveal the relationship between data selection and uncertainty estimation of selected data. Although large language models (LLMs) exhibit exceptional capabilities in language understanding and generation, which provide new ways to alleviate data scarcity, evaluating data usability remains a challenging task. This makes efficient data selection indispensable. To mitigate these issues, we propose Entropy-Based Unsupervised Data Selection (EUDS) framework. Empirical experiments on sentiment analysis (SA), topic classification (Topic-CLS), and question answering (Q&A) tasks validate its effectiveness. EUDS establishes a computationally efficient data-filtering mechanism. Theoretical analysis and experimental results confirm the effectiveness of our approach. EUDS significantly reduces computational costs and improves training time efficiency with less data requirement. This provides an innovative solution for the efficient fine-tuning of LMs in the compute-constrained scenarios.

💡 Research Summary

The paper addresses the growing resource bottlenecks in fine‑tuning modern language models (LMs), namely the need for large computational budgets and massive amounts of high‑quality data. While data‑selection techniques have been proposed to reduce the training set size, many of them are computationally expensive themselves, making them unsuitable for compute‑constrained scenarios. To bridge this gap, the authors introduce the Entropy‑Based Unsupervised Data Selection (EUDS) framework, which filters both human‑generated and synthetic data before fine‑tuning, thereby remaining model‑agnostic and inexpensive.

EUDS relies on three complementary entropy measures: (1) Information Entropy (IE) computed over n‑gram distributions (unigrams, bigrams, trigrams) with adjustable weights, capturing lexical diversity; (2) Generative Entropy (GE) derived from the average log‑probability (perplexity) of a pretrained LM when generating each token, reflecting predictive difficulty; and (3) Semantic Entropy (SE) obtained by clustering semantically equivalent generation outcomes and measuring the entropy of the resulting class distribution, thus focusing on true semantic uncertainty while ignoring surface‑form variation.

Rather than exhaustively searching the entire candidate pool, EUDS adopts an interval‑based selection strategy. A representative random subset of the full dataset is first sampled. For each sample in this subset, the three entropy scores are computed and the samples are partitioned into quantile‑based intervals. Each interval is then evaluated by fine‑tuning a lightweight model on the interval’s data and measuring validation performance. The interval(s) that achieve the best trade‑off between data reduction and accuracy are identified as the optimal entropy range. This optimal range is then applied to the whole candidate set, selecting all samples whose entropy scores fall within it. Because the relationship between entropy and downstream performance is assumed to generalize from the subset to the full set, the method avoids costly global searches while still preserving the performance‑data relationship.

Synthetic data generation is performed with GPT‑4o using few‑shot prompting and temperature control to produce diverse, label‑conditioned texts for three downstream tasks: sentiment analysis (SA), topic classification (Topic‑CLS), and question answering (Q&A). The synthetic data are treated exactly like original data in the entropy calculations, enabling a unified selection pipeline that can mix both sources. Experiments show that, after EUDS filtering, the combined original‑synthetic dataset can be reduced by 40‑70 % without any loss in standard metrics (accuracy, F1, exact match). In many cases, the filtered set even yields modest gains, likely because low‑entropy (redundant) or overly noisy high‑entropy samples are removed.

From a computational standpoint, entropy computation scales linearly with text length (O(N)), and the interval selection requires only a single lightweight fine‑tuning run on the subset, making EUDS 3‑5× faster than influence‑based or gradient‑based selection methods. The framework is model‑agnostic: it can be placed before any fine‑tuning pipeline, whether the downstream model is BERT, RoBERTa, GPT‑2, or a newer transformer, because it does not modify the model architecture or require internal gradients.

The authors also explore combined entropy strategies, such as weighted averages of IE, GE, and SE or multi‑objective optimization, demonstrating that a composite score can better balance lexical richness, generation difficulty, and semantic clarity, especially when synthetic data dominate the pool. Large‑scale validation confirms that even a single‑entropy variant (e.g., GE alone) maintains baseline performance, underscoring the scalability of the approach.

In summary, EUDS contributes three key innovations: (1) a multi‑level entropy quantification of data uncertainty, (2) a subset‑driven interval selection that efficiently identifies the most informative entropy range, and (3) a unified pipeline that seamlessly integrates synthetic data generated by LLMs. Together, these enable effective fine‑tuning of language models under strict compute budgets, reduce data acquisition costs, and open avenues for privacy‑preserving or low‑resource scenarios. Future work is suggested on incorporating additional model‑internal signals (e.g., attention patterns) and extending the method to multimodal datasets.