Feature Engineering is Not Dead: Reviving Classical Machine Learning with Entropy, HOG, and LBP Feature Fusion for Image Classification

Feature engineering continues to play a critical role in image classification, particularly when interpretability and computational efficiency are prioritized over deep learning models with millions of parameters. In this study, we revisit classical machine learning based image classification through a novel approach centered on Permutation Entropy (PE), a robust and computationally lightweight measure traditionally used in time series analysis but rarely applied to image data. We extend PE to two-dimensional images and propose a multiscale, multi-orientation entropy-based feature extraction approach that characterizes spatial order and complexity along rows, columns, diagonals, anti-diagonals, and local patches of the image. To enhance the discriminatory power of the entropy features, we integrate two classic image descriptors: the Histogram of Oriented Gradients (HOG) to capture shape and edge structure, and Local Binary Patterns (LBP) to encode micro-texture of an image. The resulting hand-crafted feature set, comprising of 780 dimensions, is used to train Support Vector Machine (SVM) classifiers optimized through grid search. The proposed approach is evaluated on multiple benchmark datasets, including Fashion-MNIST, KMNIST, EMNIST, and CIFAR-10, where it delivers competitive classification performance without relying on deep architectures. Our results demonstrate that the fusion of PE with HOG and LBP provides a compact, interpretable, and effective alternative to computationally expensive and limited interpretable deep learning models. This shows a potential of entropy-based descriptors in image classification and contributes a lightweight and generalizable solution to interpretable machine learning in image classification and computer vision.

💡 Research Summary

The paper revisits classical feature‑engineering approaches for image classification, arguing that they remain valuable when interpretability and computational efficiency are prioritized over the massive parameter counts of modern deep networks. The authors introduce a novel adaptation of Permutation Entropy (PE)—originally a 1‑D time‑series complexity measure—to two‑dimensional images. By treating each row, column, diagonal, anti‑diagonal, and local patch as a 1‑D signal, they compute forward and reversed PE, combine them via a geometric mean, and thus obtain a scalar complexity value per direction. This multi‑orientation, multi‑scale scheme yields a set of PE features that capture spatial order and irregularity across the whole image.

Because PE alone does not encode shape or fine‑grained texture, the authors fuse it with two well‑established handcrafted descriptors: Histogram of Oriented Gradients (HOG) for global edge and shape information, and Local Binary Patterns (LBP) for micro‑texture and illumination‑invariant patterns. HOG is extracted using a dense grid of 8×8 cells with 9 orientation bins, while LBP uses an 8‑neighbour circular pattern. The three groups are concatenated into a 780‑dimensional feature vector (approximately 60 dimensions from PE, 360 from HOG, and 360 from LBP, the exact split varying slightly with image size). Each component retains a clear semantic meaning, making the overall representation highly interpretable.

For classification, Support Vector Machines (SVM) are employed. The authors perform an exhaustive grid search over kernel types (linear, polynomial, RBF) and hyper‑parameters (C, γ) for each dataset, selecting the configuration that maximizes validation accuracy. Experiments are conducted on four widely used benchmarks: Fashion‑MNIST, KMNIST, EMNIST, and CIFAR‑10. The proposed PE‑HOG‑LBP pipeline achieves 91.23 % test accuracy on Fashion‑MNIST, surpassing the previously reported best non‑deep learning result of 89.7 %. On the other three datasets, accuracies range from 84 % to 88 %, which is competitive with many shallow models and approaches the performance of lightweight deep nets while using far fewer resources.

Beyond raw accuracy, the authors emphasize computational advantages. The entire feature extraction and SVM inference can be performed on a standard CPU in a few seconds per dataset, requiring orders of magnitude less memory and energy than convolutional networks that need GPUs and extensive training epochs. The code, along with preprocessing scripts, is released on GitHub, ensuring reproducibility and facilitating future extensions to domain‑specific high‑resolution data such as medical scans or satellite imagery.

The paper’s contributions can be summarized as follows: (1) a 2‑D, multi‑orientation, multi‑scale formulation of Permutation Entropy for image complexity analysis; (2) a principled fusion of PE with HOG and LBP that yields a compact, semantically rich feature vector; (3) empirical evidence that a classical SVM classifier, when fed these handcrafted features, can match or exceed state‑of‑the‑art shallow methods and approach deep‑learning baselines; (4) a demonstration of the interpretability and efficiency benefits of feature‑engineering‑centric pipelines, reinforcing the claim that “feature engineering is not dead.” The work provides a solid baseline for researchers seeking lightweight, explainable alternatives to deep learning in image classification tasks.