Evaluating Graphical Perception Capabilities of Vision Transformers

Vision Transformers, ViTs, have emerged as a powerful alternative to convolutional neural networks, CNNs, in a variety of image-based tasks. While CNNs have previously been evaluated for their ability to perform graphical perception tasks, which are essential for interpreting visualizations, the perceptual capabilities of ViTs remain largely unexplored. In this work, we investigate the performance of ViTs in elementary visual judgment tasks inspired by the foundational studies of Cleveland and McGill, which quantified the accuracy of human perception across different visual encodings. Inspired by their study, we benchmark ViTs against CNNs and human participants in a series of controlled graphical perception tasks. Our results reveal that, although ViTs demonstrate strong performance in general vision tasks, their alignment with human-like graphical perception in the visualization domain is limited. This study highlights key perceptual gaps and points to important considerations for the application of ViTs in visualization systems and graphical perceptual modeling.

💡 Research Summary

The paper investigates whether modern Vision Transformers (ViTs) possess the low‑level graphical perception abilities that are fundamental to data visualization, a capability that has previously been examined for Convolutional Neural Networks (CNNs) but not for ViTs. Drawing on the classic Cleveland‑McGill hierarchy of visual encodings, the authors design a suite of nine elementary perception tasks (position on a common scale, position on non‑aligned scales, length, direction, angle, area, volume, curvature, and color contrast) and focus on seven of them that are most relevant to chart interpretation.

Three representative ViT architectures are evaluated: the vanilla Vision Transformer (vViT), the Convolutional Vision Transformer (CvT), and the Shifted Window Transformer (Swin). These models were pre‑trained on ImageNet and then fine‑tuned on a synthetic dataset containing 10 000 images per encoding, ensuring that each model sees the same visual stimuli. For comparison, two state‑of‑the‑art CNNs (ResNet‑50 and EfficientNet‑B3) are included, and a human baseline is obtained from 48 participants with diverse backgrounds.

Performance is measured using absolute error, root‑mean‑square error (RMSE), and reaction time (RT). Human baselines are reported as mean error with 95 % confidence intervals. The results reveal a nuanced picture. On position (common scale) and length tasks, all ViT variants achieve errors comparable to humans (average error ≈ 2 % versus the human mean of 1.9 %). However, for tasks that require non‑linear transformations—particularly angle and area—ViTs exhibit substantially larger errors. Swin performs worst on angle estimation (average error ≈ 12.4 %), while CvT shows relatively better area estimation (≈ 7.8 % error) but still lags behind human performance. Volume and curvature tasks, though limited by data, follow the same trend: ViTs are less accurate than humans, and CNNs often provide slightly more stable results. All models respond faster than humans (3–5× quicker), highlighting a speed‑accuracy trade‑off rooted in algorithmic processing versus human cognition.

The authors interpret these findings through two lenses. First, the self‑attention mechanism of ViTs excels at capturing global relationships but does not replicate the hierarchical, context‑driven attention that humans employ when interpreting basic visual encodings. Second, the fine‑tuning data lack the breadth of real‑world visual variations (e.g., rotations, scaling, occlusions) that humans encounter, limiting the models’ ability to generalize to perceptual judgments. Consequently, while ViTs are powerful for high‑level vision tasks, their alignment with human perceptual hierarchies remains limited, raising concerns for applications such as automated chart analysis, design recommendation systems, and visual question answering where perceptual fidelity is critical.

To bridge this gap, the paper proposes several future research directions: (1) incorporating attention regularization schemes that mimic human selective attention, (2) constructing large‑scale, low‑level encoding‑specific datasets for more targeted fine‑tuning, (3) leveraging multimodal training (vision‑language models) to embed semantic context that may aid perceptual reasoning, and (4) designing interactive human‑in‑the‑loop frameworks that allow models to receive corrective feedback on perceptual errors. By pursuing these avenues, the community can move toward vision models that not only achieve state‑of‑the‑art performance on benchmark tasks but also faithfully reproduce the perceptual judgments that underlie effective data visualization.