Automatic Detection and Analysis of Singing Mistakes for Music Pedagogy

The advancement of machine learning in audio analysis has opened new possibilities for technology-enhanced music education. This paper introduces a framework for automatic singing mistake detection in the context of music pedagogy, supported by a newly curated dataset. The dataset comprises synchronized teacher learner vocal recordings, with annotations marking different types of mistakes made by learners. Using this dataset, we develop different deep learning models for mistake detection and benchmark them. To compare the efficacy of mistake detection systems, a new evaluation methodology is proposed. Experiments indicate that the proposed learning-based methods are superior to rule-based methods. A systematic study of errors and a cross-teacher study reveal insights into music pedagogy that can be utilised for various music applications. This work sets out new directions of research in music pedagogy. The codes and dataset are publicly available.

💡 Research Summary

**
The paper addresses a gap in music education technology by proposing an automatic singing‑mistake detection framework specifically designed for Indian Art Music (IAM) pedagogy. The authors first construct a novel dataset, named M3 (MADHA V Lab Mistake Detection for Music Teaching Database), which consists of synchronized teacher‑learner vocal recordings from two expert teachers and a large number of beginner learners. Each learner recording is annotated by the corresponding teacher with precise start and end times for four mistake categories: frequency (pitch), amplitude (loudness), pronunciation, and timing. The annotations are frame‑wise binary labels, enabling a multi‑label detection task.

To prepare the audio for modeling, the authors extract pitch contours using Praat‑Parselmouth with a 60 ms window and 10 ms hop, then normalize the pitch by the tonic of each piece to obtain an octave‑invariant log‑pitch representation. Unvoiced frames are assigned a sentinel value of –1 to separate silence from valid pitch. For amplitude mistakes, short‑time RMS energy is computed, log‑compressed, and used as a feature. Because amplitude errors are rare, the authors augment the training data by artificially modifying the energy of stable note segments, thereby mitigating class imbalance.

Three deep learning architectures are evaluated: a 1‑D Convolutional Neural Network (CNN), a Convolutional Recurrent Neural Network (CRNN) that adds bidirectional GRU layers after convolutional blocks, and a Temporal Convolutional Network (TCN) that stacks causal dilated convolutions. Input features combine the log‑pitch (or chromagram) and log‑RMS channels, and the models output two binary streams corresponding to frequency and amplitude mistakes. Binary cross‑entropy loss is applied per label, with class‑weighting to emphasize the minority amplitude class.

The authors also introduce a task‑specific evaluation methodology that goes beyond simple frame‑level accuracy. They compute precision, recall, and F1‑score for each mistake type, and propose a composite metric that balances detection quality across both categories. Additionally, a cross‑teacher analysis is performed to examine how different teaching styles influence the distribution of mistakes.

Experimental results show that all deep learning models outperform a rule‑based baseline that relies on simple pitch deviation thresholds and RMS thresholds. The CRNN achieves the highest F1‑score, improving over the rule‑based system by roughly 12 % for frequency mistakes and by a larger margin for amplitude mistakes due to its ability to capture temporal dependencies. The TCN offers comparable performance with lower computational overhead, making it suitable for real‑time deployment.

Beyond quantitative performance, the study provides pedagogical insights. The cross‑teacher analysis reveals that one teacher’s recordings lead to more frequent timing errors, while the other’s result in higher pitch‑related errors, suggesting that teacher‑specific feedback strategies could be tailored based on detected error patterns. The authors argue that fine‑grained mistake detection can give learners immediate, actionable feedback, bridging the gap caused by the traditional guru‑shishya model where feedback is limited to in‑person sessions.

All code, pre‑trained models, and the M3 dataset are released publicly, ensuring reproducibility and encouraging further research. The authors discuss future directions such as extending the framework to real‑time feedback applications, incorporating additional mistake categories (e.g., ornamentation, expressive timing), and adapting the system to other musical traditions that rely on oral transmission.

In summary, the paper makes four key contributions: (1) formalizing automatic singing‑mistake detection as a multi‑label, frame‑wise classification problem; (2) providing the first publicly available, teacher‑annotated IAM dataset for this purpose; (3) developing and benchmarking CNN, CRNN, and TCN models that demonstrably outperform rule‑based approaches; and (4) offering a new evaluation protocol and pedagogical analysis that together lay the groundwork for intelligent, data‑driven music tutoring systems.