Modeling Music Modality with a Key-Class Invariant Pitch Chroma CNN

This paper presents a convolutional neural network (CNN) that uses input from a polyphonic pitch estimation system to predict perceived minor/major modality in music audio. The pitch activation input is structured to allow the first CNN layer to compute two pitch chromas focused on different octaves. The following layers perform harmony analysis across chroma and time scales. Through max pooling across pitch, the CNN becomes invariant with regards to the key class (i.e., key disregarding mode) of the music. A multilayer perceptron combines the modality activation output with spectral features for the final prediction. The study uses a dataset of 203 excerpts rated by around 20 listeners each, a small challenging data size requiring a carefully designed parameter sharing. With an R2 of about 0.71, the system clearly outperforms previous systems as well as individual human listeners. A final ablation study highlights the importance of using pitch activations processed across longer time scales, and using pooling to facilitate invariance with regards to the key class.

💡 Research Summary

The paper introduces a compact convolutional neural network (CNN) specifically designed to predict perceived minor/major modality in music audio, using high‑resolution polyphonic pitch activations as its sole harmonic input. The authors start from a state‑of‑the‑art pitch‑tracking system that produces a “Pitchogram” with 1‑cent frequency resolution and 5.8 ms temporal resolution. After smoothing and down‑sampling to one semitone per bin, the pitch representation is globally retuned to compensate for non‑standard concert pitch, then limited to MIDI notes 26–96, yielding a 71‑dimensional vector per frame.

To expose both pitch class and octave information, each frame is split into five overlapping 23‑semitone sections spaced an octave apart, and these sections are stacked along a depth dimension. Simultaneously, the authors generate six versions of the pitch vector at different temporal scales by convolving with Hann windows of widths ranging from 31 frames (≈0.18 s) to 2431 frames (≈14.1 s). The six scale‑specific vectors are concatenated, producing a three‑dimensional tensor of shape 23 (pitch class) × 6 (time scale) × 5 (octave) for every time step.

The CNN processes each 9‑second segment (≈1550 frames) independently. Its architecture is deliberately tiny—only 413 learnable parameters—including batch‑normalization after each ReLU. The first “chroma” layer contains two filters (one per octave) that convert the raw pitch tensor into two pitch‑chroma maps. These maps are fed into a “harmony analysis” layer with one filter for the first chroma and five filters for the second, allowing the network to treat bass and higher registers differently while keeping parameter count low. The resulting feature maps are concatenated and passed through a set of six filters that span all pitch classes and integrate information across the six time scales.

Key to the design is a max‑pooling operation that spans the full 12 pitch‑class dimension. By taking the maximum response across all keys, the network becomes invariant to the absolute key (the “key class”) of the excerpt; it only retains the strongest evidence for either minor or major harmonic patterns, regardless of transposition. The pooled activations are then fed into a locally fully‑connected layer implemented as seven filters that cover the entire spatial extent of a single frame, followed by a single filter that produces a frame‑level regression output. Finally, average pooling across all frames of the segment yields a segment‑level modality prediction.

Training is performed with 10‑fold cross‑validation on a modest dataset of 203 music excerpts, each rated by roughly 20 listeners on a continuous modality scale. The optimizer is Adam (initial LR 0.01, decay 0.98 per epoch), loss is mean‑squared error, L2 regularization 1e‑4, and batch size 32. Because the data are scarce, the authors enforce a quality check: if after 25 epochs the training R² is below 0.83, the network is re‑initialized and trained again (occurs in ~3 % of cases). No early‑stopping on a validation set is used, following prior work that argues small validation splits are unreliable.

Performance is compared against two prior approaches: (i) a partial‑least‑squares model using hand‑crafted modality and spectral features (R² ≈ 0.43–0.53) and (ii) an Inception‑v3 model applied to mel‑spectrograms (R² ≈ 0.23). The proposed CNN achieves an average R² of about 0.71, clearly surpassing both baselines and also outperforming the average individual human listener.

An extensive ablation study isolates the contributions of the design choices. Removing the multi‑scale temporal smoothing drops R² dramatically, confirming that listeners rely on harmonic information at various time horizons. Eliminating the key‑invariant max‑pooling also leads to a steep performance decline, demonstrating that transposition invariance is essential for generalization. Adding spectral features via an ensemble of multilayer perceptrons yields a modest further gain, indicating that timbral cues provide complementary information but are not the primary driver of modality perception.

In summary, the paper’s key contributions are: (1) a novel 3‑D pitch‑tensor representation that simultaneously encodes pitch class, octave, and multiple temporal contexts; (2) a key‑class invariant max‑pooling mechanism that forces the network to focus on relative minor/major harmonic patterns rather than absolute pitch; (3) an ultra‑compact CNN architecture that can be trained effectively on a very small dataset without overfitting. The work demonstrates that carefully engineered pitch‑based deep learning models can capture subtle aspects of musical harmony and outperform both traditional feature‑based methods and larger generic CNNs, opening avenues for MIR tasks where harmonic understanding is paramount but annotated data are limited.