Fake News Detection: It's All in the Data!

This comprehensive survey serves as an indispensable resource for researchers embarking on the journey of fake news detection. By highlighting the pivotal role of dataset quality and diversity, it underscores the significance of these elements in the effectiveness and robustness of detection models. The survey meticulously outlines the key features of datasets, various labeling systems employed, and prevalent biases that can impact model performance. Additionally, it addresses critical ethical issues and best practices, offering a thorough overview of the current state of available datasets. Our contribution to this field is further enriched by the provision of GitHub repository, which consolidates publicly accessible datasets into a single, user-friendly portal. This repository is designed to facilitate and stimulate further research and development efforts aimed at combating the pervasive issue of fake news.

💡 Research Summary

The paper presents a comprehensive survey of publicly available datasets for fake‑news detection, arguing that dataset quality and diversity are the primary determinants of model effectiveness and robustness. After an introductory motivation that highlights the societal, political, and cybersecurity harms caused by misinformation, the authors describe a GitHub repository they have built to aggregate and curate existing resources. The survey categorises datasets into four modalities: textual (e.g., LIAR, MisInfoText), visual (e.g., Verification Corpus, FCV‑2018), multimodal (e.g., FakeNewsNet, r/fakeddit) and generative‑text (e.g., M4). For each modality it details typical features—linguistic cues such as n‑grams, part‑of‑speech tags, sentiment scores; metadata like timestamps, author information, and social‑engagement metrics; visual cues including pixel‑level manipulation traces—and explains how these are used by machine‑learning pipelines.

A substantial portion of the work is devoted to labeling schemes. The authors enumerate binary labels, three‑point scales (PHEME), five‑point credibility scales (CREDBANK), and six‑point truthfulness scales (LIAR), illustrating the trade‑offs between annotation granularity, annotator effort, and model interpretability. They provide two summary tables: one comparing rating scales across datasets, and another summarising year of release, language coverage, modality, and accessibility.

The impact of dataset properties on detection algorithms is examined empirically. Larger, well‑annotated corpora such as NELA‑GT‑2018 improve generalisation, whereas small or highly imbalanced collections (e.g., Verification Corpus) tend to cause over‑fitting. Balanced class distributions, realistic annotation processes, and inclusion of diverse feature types consistently boost performance across a range of classifiers.

The authors then discuss pervasive challenges: data imbalance between real and fake items, noisy user‑generated content, and the rapid evolution of misinformation that leads to distribution drift. They identify three major bias sources—selection bias (over‑representation of certain fake‑news topics), labeling bias (inconsistent or subjective annotations), and cultural/linguistic bias (datasets dominated by English or a single cultural context). These biases can degrade accuracy, cause inconsistent results across domains, and limit cross‑lingual transferability.

A dedicated section highlights the superiority of multimodal datasets. By jointly modelling text and visual signals, multimodal approaches achieve 8–11 % higher accuracy than text‑only baselines; the paper cites the Fakeddit dataset achieving 87 % accuracy with a CNN‑based multimodal architecture. However, the creation of multimodal corpora is resource‑intensive, requiring synchronized collection, complex annotation pipelines, and careful quality control.

Finally, the survey outlines future directions: (1) establishing systematic, version‑controlled updates to keep datasets current with emerging misinformation trends; (2) expanding multilingual and multicultural coverage to improve global applicability; (3) incorporating synthetic, AI‑generated misinformation to anticipate next‑generation threats; and (4) formalising ethical guidelines for data collection and annotation to protect privacy and reduce annotator bias. In sum, the paper argues convincingly that a data‑centric perspective—emphasising high‑quality, diverse, and ethically curated datasets—is essential for building reliable, adaptable fake‑news detection systems.