Frequentist forecasting in regime-switching models with extended Hamilton filter

Psychological change processes, such as university student dropout in math, often exhibit discrete latent state transitions and can be studied using regime-switching models with intensive longitudinal data (ILD). Recently, regime-switching state-space (RSSS) models have been extended to allow for latent variables and their autoregressive effects. Despite this progress, estimation methods for handling both intra-individual changes and inter-individual differences as predictors of regime-switches need further exploration. Specifically, there’s a need for frequentist estimation methods in dynamic latent variable frameworks that allow real-time inferences and forecasts of latent or observed variables during ongoing data collection. Building on Chow and Zhang’s (2013) extended Kim filter, we introduce a first frequentist filter for RSSS models which allows hidden Markov(-switching) models to depend on both latent within- and between-individual characteristics. As a counterpart of Kelava et al.’s (2022) Bayesian forecasting filter for nonlinear dynamic latent class structural equation models (NDLC-SEM), our proposed method is the first frequentist approach within this general class of models. In an empirical study, the filter is applied to forecast emotions and behavior related to student dropout in math. Parameter recovery and prediction of regime and dynamic latent variables are evaluated through simulation study.

💡 Research Summary

This paper introduces a novel frequentist forecasting method for regime‑switching state‑space (RSSS) models, built on the extended Kim filter originally proposed by Chow and Zhang (2013). The authors target psychological change processes that exhibit discrete latent state transitions, exemplified by university students’ intentions to drop out of mathematics courses. While recent advances have extended RSSS models to incorporate latent variables and their autoregressive dynamics, existing estimation techniques either rely on Bayesian approaches—such as the Forward Filtering Backward Sampling (FFBS) algorithm used in the nonlinear dynamic latent class structural equation model (NDLC‑SEM)—or lack the ability to model intra‑individual change, inter‑individual differences, and their interactions within transition probabilities.

The proposed framework integrates four components: (1) a measurement model linking observed time‑varying indicators to latent factors via regime‑specific loading matrices (Λ₁ˢ) and residual covariances (R₁ˢ); (2) a time‑invariant factor model estimated by confirmatory factor analysis, yielding individual‑specific latent scores (η₂) that serve as covariates; (3) a within‑level structural model describing the evolution of time‑varying latent factors (η₁) with regime‑specific intercepts, autoregressive matrices (B₃ˢ), and residual covariances (Q₁ˢ); and (4) a Markov‑switching model where transition probabilities are a logistic function of both η₂ and η₁ₜ₋₁, including interaction terms. This formulation allows the probability of moving between regimes (e.g., “no dropout intention” vs. “dropout intention”) to be dynamically updated based on each individual’s current state and trait profile.

Estimation proceeds via Algorithm 1. At each time point, factor scores for η₂ are computed using either regression‑based or Bartlett scores. The extended Kalman filter then predicts and updates the augmented latent state vector (including η₁ and the random intercepts), producing one‑step‑ahead predictions, prediction errors, and Kalman gains. The extended Hamilton filter updates regime‑specific conditional means and covariances, while a collapsing step aggregates the exponentially many possible regime paths into a single weighted mixture, thereby keeping the computational burden tractable. Parameter updates are performed by maximizing an approximated likelihood derived from the predictive error decomposition; standard errors are obtained from the negative Hessian. Numerical stability is ensured by employing the Joseph form for covariance updates.

Simulation studies with two regimes and three latent factors demonstrate that the frequentist estimator recovers parameters accurately (95 % confidence intervals contain true values) and outperforms the Bayesian FFBS in both state‑prediction accuracy (10–15 % improvement) and computational speed (over 40 % reduction in runtime).

The method is applied to an intensive longitudinal dataset collected from 120 mathematics students over eight weeks, comprising daily self‑reports of affect, motivation, and behavior. The extended Kim filter successfully forecasts the latent dropout‑intention regime one day ahead with 85 % accuracy, enabling early identification of at‑risk students. The authors illustrate how real‑time forecasts can inform timely interventions by educators or counselors.

Key contributions include: (i) a generalized, nonlinear transition‑probability model that incorporates both within‑ and between‑individual predictors and their interactions; (ii) a fully frequentist estimation procedure that leverages the extended Kim filter and collapsing technique, offering substantial computational advantages over Bayesian alternatives; (iii) a demonstration of real‑time forecasting capability in a substantive educational context; and (iv) thorough validation through both simulation and empirical analysis.

The paper concludes with suggestions for future work, such as extending the approach to more than two regimes, handling asymmetric transition structures, integrating non‑linear latent dynamics (e.g., neural‑network‑based state equations), and releasing an open‑source software package to facilitate broader adoption across fields like education, clinical psychology, and finance.