Energy-Efficient Information Representation in MNIST Classification Using Biologically Inspired Learning

Efficient representation learning is essential for optimal information storage and classification. However, it is frequently overlooked in artificial neural networks (ANNs). This neglect results in networks that can become overparameterized by factors of up to 13, increasing redundancy and energy consumption. As the demand for large language models (LLMs) and their scale increase, these issues are further highlighted, raising significant ethical and environmental concerns. We analyze our previously developed biologically inspired learning rule using information-theoretic concepts, evaluating its efficiency on the MNIST classification task. The proposed rule, which emulates the brain’s structural plasticity, naturally prevents overparameterization by optimizing synaptic usage and retaining only the essential number of synapses. Furthermore, it outperforms backpropagation (BP) in terms of efficiency and storage capacity. It also eliminates the need for pre-optimization of network architecture, enhances adaptability, and reflects the brain’s ability to reserve ‘space’ for new memories. This approach advances scalable and energy-efficient AI and provides a promising framework for developing brain-inspired models that optimize resource allocation and adaptability.

💡 Research Summary

The paper tackles the pervasive problem of over‑parameterization and associated energy waste in modern artificial neural networks by introducing a biologically inspired learning rule that mimics the brain’s structural plasticity. The authors argue that conventional deep learning, which relies almost exclusively on weight‑based plasticity and gradient descent, inevitably activates every synapse, leading to redundant parameters—sometimes up to 13‑fold excess—and inflated carbon footprints, especially as large language models continue to scale.

Building on their earlier work, they propose a learning framework that combines three mechanisms: (i) competitive excitatory Hebbian plasticity, (ii) non‑negativity constraints on synaptic weights, and (iii) a weight‑perturbation (WP) component that adjusts weights based on the difference between perturbed and unperturbed error signals. The hidden‑layer update (Equation 1) follows a Hebbian rule, while the output‑layer update (Equation 2) blends Hebbian changes with WP adjustments, weighted by hyper‑parameters α and β. A bias‑update rule (Equation 4) implements homeostatic control. This combination encourages the network to retain only those synapses that contribute meaningfully to the task, effectively pruning silent connections without an explicit pruning stage.

To evaluate the efficiency of information representation, the authors adopt an information‑theoretic perspective inspired by Tishby and Zaslavsky’s “information bottleneck” framework. They treat each layer as a random variable in a Markov chain and estimate the mutual information I(X;Z) between the input X and a stochastic latent variable Z using a variational auto‑encoder (VAE) based approach. Importantly, they freeze a deterministic hidden representation H (extracted from a pretrained network) and attach a stochastic encoder that outputs mean µ and variance σ, thereby approximating p(Z|H) as a Gaussian. Mutual information is then computed via the KL‑divergence D_KL