Real-Time Human Activity Recognition on Edge Microcontrollers: Dynamic Hierarchical Inference with Multi-Spectral Sensor Fusion

The demand for accurate on-device pattern recognition in edge applications is intensifying, yet existing approaches struggle to reconcile accuracy with computational constraints. To address this challenge, a resource-aware hierarchical network based on multi-spectral fusion and interpretable modules, namely the Hierarchical Parallel Pseudo-image Enhancement Fusion Network (HPPI-Net), is proposed for real-time, on-device Human Activity Recognition (HAR). Deployed on an ARM Cortex-M4 microcontroller for low-power real-time inference, HPPI-Net achieves 96.70% accuracy while utilizing only 22.3 KiB of RAM and 439.5 KiB of ROM after optimization. HPPI-Net employs a two-layer architecture. The first layer extracts preliminary features using Fast Fourier Transform (FFT) spectrograms, while the second layer selectively activates either a dedicated module for stationary activity recognition or a parallel LSTM-MobileNet network (PLMN) for dynamic states. PLMN fuses FFT, Wavelet, and Gabor spectrograms through three parallel LSTM encoders and refines the concatenated features using Efficient Channel Attention (ECA) and Depthwise Separable Convolution (DSC), thereby offering channel-level interpretability while substantially reducing multiply-accumulate operations. Compared with MobileNetV3, HPPI-Net improves accuracy by 1.22% and reduces RAM usage by 71.2% and ROM usage by 42.1%. These results demonstrate that HPPI-Net achieves a favorable accuracy-efficiency trade-off and provides explainable predictions, establishing a practical solution for wearable, industrial, and smart home HAR on memory-constrained edge platforms.

💡 Research Summary

The paper introduces HPPI‑Net (Hierarchical Parallel Pseudo‑image Enhancement Fusion Network), a resource‑aware deep learning architecture designed for real‑time human activity recognition (HAR) on ultra‑low‑power microcontrollers. The authors target the ARM Cortex‑M4 platform, where memory (tens of KiB) and compute (few hundred K‑MACs) are extremely limited, yet many wearable and industrial applications demand on‑device inference with high accuracy.

HPPI‑Net adopts a two‑stage hierarchical inference strategy. In the first stage, raw 6‑axis IMU data (three‑axis accelerometer and three‑axis gyroscope) are segmented into non‑overlapping windows of 16 samples. Fast Fourier Transform (FFT) spectrograms are generated for each window and fed into a lightweight CNN‑LSTM module that quickly classifies the window into one of three coarse categories: Moving, Stationary, or Cycling. This stage consumes only ~2 KB of RAM and provides a rapid, low‑cost decision that determines which second‑stage branch to activate.

If the coarse decision indicates a stationary activity, the same CNN‑LSTM from stage‑one is reused for fine‑grained classification, avoiding any additional computation. For dynamic activities, a dedicated Parallel LSTM‑MobileNet Network (PLMN) is invoked. PLMN processes three complementary time‑frequency representations—FFT, Wavelet Transform (WT), and Gabor Transform (GT)—each treated as a pseudo‑image (16 × 6). Three parallel LSTM encoders extract temporal features from each spectral view. Their outputs are concatenated along the channel dimension and passed through an Efficient Channel Attention (ECA) block, which learns per‑channel importance without heavy parameter overhead. The fused representation is then refined by Depthwise Separable Convolution (DSC) blocks, borrowing the computational efficiency of MobileNet while discarding the full MobileNet backbone.

Key technical contributions include:

1. Conditional hierarchical execution – By activating the heavy PLMN only for dynamic activities, the model reduces average RAM usage by 71 % and MAC count by a comparable margin.
2. Multi‑spectral fusion – FFT captures global periodicity, WT excels at transient changes, and GT provides high time‑frequency localization. Their parallel LSTM encoders preserve temporal ordering before fusion, yielding richer discriminative features than any single transform.
3. Lightweight attention and convolution – ECA supplies channel‑level interpretability (the model can highlight which spectral branch or sensor axis contributed most to a decision) while adding negligible compute. DSC dramatically cuts parameters and MACs relative to standard convolutions, making the network MCU‑friendly.
4. Full‑stack MCU deployment – The authors quantize the network to 8‑bit integers, apply operator fusion, and compile it with TensorFlow Lite Micro into pure C code. On a Cortex‑M4 (STM32F4) the final binary occupies 439.5 KiB ROM and runs inference in under 2 ms per window, satisfying real‑time constraints.

Experimental evaluation involved 20 healthy volunteers performing seven activities (walking, running, stair ascent/descent, standing, lying down, cycling) while wearing a wrist‑mounted 6‑axis IMU sampled at 50 Hz. Data were pre‑processed with a 3‑point median filter and segmented as described. The authors compared HPPI‑Net against a baseline MobileNetV3 and a single‑spectral LSTM‑CNN model. Results:

• Accuracy: HPPI‑Net 96.70 % vs MobileNetV3 95.48 % (Δ +1.22 %).
• RAM: 22.3 KiB vs 77.5 KiB (‑71.2 %).
• ROM: 439.5 KiB vs 758 KiB (‑42.1 %).
• Inference latency: ≤ 1.8 ms per window on the MCU.

Post‑hoc analysis using an MLP‑based feature attribution method confirmed that different spectral branches dominate different activities (e.g., WT for stair transitions, GT for cycling). This provides the explainability required for safety‑critical domains such as healthcare monitoring.

In conclusion, HPPI‑Net successfully balances three critical dimensions—high classification performance, ultra‑low resource consumption, and interpretability—making it a practical solution for edge‑deployed HAR in wearables, smart homes, and industrial IoT. The paper suggests future work on extending the architecture to additional sensor modalities, exploring event‑driven data streams, and co‑designing hardware accelerators to further push the limits of on‑device intelligence.