A Joint Model for Multimodal Document Quality Assessment

The quality of a document is affected by various factors, including grammaticality, readability, stylistics, and expertise depth, making the task of document quality assessment a complex one. In this paper, we explore this task in the context of assessing the quality of Wikipedia articles and academic papers. Observing that the visual rendering of a document can capture implicit quality indicators that are not present in the document text — such as images, font choices, and visual layout — we propose a joint model that combines the text content with a visual rendering of the document for document quality assessment. Experimental results over two datasets reveal that textual and visual features are complementary, achieving state-of-the-art results.

💡 Research Summary

The paper “A Joint Model for Multimodal Document Quality Assessment” tackles the challenging problem of automatically evaluating the quality of documents, focusing on two distinct domains: Wikipedia articles and academic papers from arXiv. Traditional approaches to document quality assessment have relied almost exclusively on textual cues—such as length, heading density, readability scores, and metadata about editors—or on handcrafted features. The authors argue that the visual rendering of a document, which includes layout, images, infoboxes, font choices, and other design elements, contains implicit quality signals that are not captured by text alone. To exploit this insight, they propose a multimodal architecture that jointly learns from both the raw text and a screenshot of the rendered document.

The visual component uses a pre‑trained Inception V3 convolutional neural network (CNN), originally trained on ImageNet, and fine‑tunes it on screenshots of documents. Each document is converted into a 1,000 × 2,000 pixel image using PhantomJS (for Wikipedia) or ImageMagick (for arXiv PDFs). Care is taken to strip any explicit quality markers (e.g., the “Featured Article” star icon) so that the model must infer quality from visual structure alone. The textual component follows the hierarchical bi‑directional LSTM (biLSTM) design introduced by Shen, Qi, and Baldwin (2017). Words are embedded, sentences are obtained via average pooling of word embeddings, and a biLSTM processes the sequence of sentence vectors to produce a document‑level representation; a final max‑pooling layer selects the most salient sentence‑level features.

The two modality‑specific embeddings are concatenated and fed into a simple feed‑forward layer followed by a softmax classifier. The entire system is trained end‑to‑end using cross‑entropy loss, with the visual and textual sub‑networks optionally pre‑trained before joint fine‑tuning.

Dataset construction is a substantial contribution. For Wikipedia, the authors crawl articles from each of the six quality classes defined by the community (Featured, Good, B, C, Start, Stub). After removing markup that directly reveals the class, they randomly sample 5,000 articles per class, yielding a balanced set of 29,794 documents. Each article is paired with a screenshot captured at the same revision, ensuring textual and visual views correspond. For arXiv, they adopt the dataset from Kang et al. (2018), which includes three computer‑science sub‑domains (AI, Computation & Language, Machine Learning). Papers are labeled “Accepted” if they appear in DBLP or were published at major conferences; otherwise they are “Rejected”. PDFs are truncated or padded to 12 pages, then rendered into a single 1,000 × 2,000 image, providing a uniform visual input.

Experimental results demonstrate that visual features alone achieve performance comparable to strong text‑only baselines. On the Wikipedia set, the visual model attains a 2.9 % absolute accuracy gain over a purely textual model when evaluated in isolation. More importantly, the joint model consistently outperforms both unimodal baselines on three of the four evaluation settings (all Wikipedia classes and two of the three arXiv subsets). The improvements confirm that visual cues—such as the presence of infoboxes, the density of images, and overall layout sophistication—provide complementary information to textual signals like readability and lexical richness.

The authors highlight three main contributions: (1) introducing visual document renderings as a novel source of quality‑related features; (2) demonstrating that a simple concatenation‑based multimodal architecture yields state‑of‑the‑art results across diverse domains; and (3) releasing large‑scale, fully paired text‑image datasets for Wikipedia and arXiv, along with code for reproducibility.

Limitations are acknowledged. Generating high‑resolution screenshots is computationally expensive and may introduce irrelevant UI elements (ads, navigation bars) that act as noise. The use of a generic image classifier (Inception V3) may not be optimal for capturing layout‑specific patterns; specialized document‑layout CNNs could further improve performance. Future work could explore attention mechanisms that dynamically weight visual versus textual contributions per document, multi‑task learning that predicts auxiliary signals (e.g., editor reputation), and real‑time integration into editing platforms to provide immediate quality feedback.

Overall, the paper convincingly shows that visual rendering is a valuable, previously under‑exploited modality for document quality assessment, and that even a straightforward multimodal fusion can substantially boost predictive accuracy in real‑world, large‑scale settings.