From error detection to behaviour observation: first results from screen capture analysis

This paper deals with errors in using spreadsheets and analysis of automatic recording of user interaction with spreadsheets. After a review of literature devoted to spreadsheet errors, we advocate the importance of going from error detection to interaction behaviour analysis. We explain how we analyze screen captures and give the main results we have obtained using this specific methodology with secondary school students (N=24). Transcription provides general characteristics: time, sequence of performed tasks, unsuccessful attempts and user preferences. Analysis reveals preferred modes of actions (toolbar buttons or menu commands), ways of writing formulas, and typical approaches in searching for solutions. Time, rhythm and density appear to be promising indicators. We think such an approach (to analyze screen captures) could be used with more advanced spreadsheet users.

💡 Research Summary

The paper investigates spreadsheet errors not merely as isolated mistakes but as observable outcomes of users’ interaction processes. After reviewing the extensive literature on spreadsheet error taxonomies—particularly the quantitative/qualitative split by Panko and Halverson and the more nuanced classifications by Rajalingham et al.—the authors argue that these frameworks, while useful for error detection, fall short of explaining how novice users generate new error patterns. They propose shifting the focus from error detection to behavior observation, treating errors as symptoms that reveal underlying cognitive and procedural issues.

To explore this perspective, the authors conducted an empirical study with 24 secondary‑school students (average age 14–15) who performed a series of spreadsheet tasks (cell formatting, formula entry, chart creation, data sorting). During task execution, every on‑screen event was captured using Camtasia Studio, a screen‑recording tool that logs mouse movements, clicks, menu selections, window changes, and keyboard inputs. A webcam simultaneously recorded the participants’ faces, providing a “vignette” of affective cues. The resulting video files were compressed and stored for later analysis.

The core methodological contribution is a detailed transcription process. Raw video streams were reviewed to identify elementary actions (e.g., “click A7”, “track to V7”) and then aggregated into higher‑level events (e.g., “select range A7:V7”). Periods of inactivity longer than ten seconds were also coded as “thinking pause”. Each event received a timestamp, and the full transcript was exported as a structured CSV file, enabling quantitative time‑series analysis.

Analysis revealed several key findings. First, tool‑selection preferences showed a strong bias toward toolbar buttons; students used toolbar commands roughly twice as often as menu items, suggesting that interface familiarity heavily influences interaction strategies. Second, formula‑writing habits were predominantly manual cell‑reference entry; advanced techniques such as absolute/relative reference mixing or the use of the fill‑handle were virtually absent, indicating limited spreadsheet proficiency among novices. Third, error types correlated with specific task phases: quantitative errors (incorrect numeric results) mainly arose during formula entry, while qualitative errors (formatting or chart‑setting mistakes) appeared during presentation tasks. Fourth, dynamic time‑based metrics—total task duration, average inter‑task interval, click‑to‑key density—proved to be strong predictors of performance. Students who quickly resumed work after an error and exhibited short pauses tended to achieve higher scores, whereas long pauses and repeated menu navigation were associated with lower achievement.

The authors discuss the implications of these metrics as proxies for cognitive load, problem‑solving strategies, and metacognitive reflection. They argue that screen‑capture combined with systematic transcription yields richer data than traditional log‑file analysis, capturing both overt actions and subtle temporal patterns. Limitations include the small, homogeneous sample and the manual nature of transcription, which introduces researcher bias. Future work is suggested to incorporate automated behavior‑recognition algorithms to reduce transcription effort and to extend the approach to diverse user groups (e.g., university students, professional analysts) to validate the generality of the identified indicators.

In conclusion, the study demonstrates that analyzing screen captures provides valuable insights into the processes that generate spreadsheet errors. Time, rhythm, and interaction density emerge as promising indicators of user competence, and the methodology offers a scalable way to assess and improve spreadsheet education and error‑prevention strategies in real‑world settings.