Learning Group Actions In Disentangled Latent Image Representations

Modeling group actions on latent representations enables controllable transformations of high-dimensional image data. Prior works applying group-theoretic priors or modeling transformations typically operate in the high-dimensional data space, where group actions apply uniformly across the entire input, making it difficult to disentangle the subspace that varies under transformations. While latent-space methods offer greater flexibility, they still require manual partitioning of latent variables into equivariant and invariant subspaces, limiting the ability to robustly learn and operate group actions within the representation space. To address this, we introduce a novel end-to-end framework that for the first time learns group actions on latent image manifolds, automatically discovering transformation-relevant structures without manual intervention. Our method uses learnable binary masks with straight-through estimation to dynamically partition latent representations into transformation-sensitive and invariant components. We formulate this within a unified optimization framework that jointly learns latent disentanglement and group transformation mappings. The framework can be seamlessly integrated with any standard encoder-decoder architecture. We validate our approach on five 2D/3D image datasets, demonstrating its ability to automatically learn disentangled latent factors for group actions in diverse data, while downstream classification tasks confirm the effectiveness of the learned representations. Our code is publicly available at https://github.com/farhanaswarnali/Learning-Group-Actions-In-Disentangled-Latent-Image-Representations .

💡 Research Summary

The paper addresses a fundamental challenge in controllable image manipulation: how to effectively model group actions (such as rotation, translation, or scaling) within latent representations without manual intervention. Traditionally, applying group-theoretic priors directly to high-dimensional pixel space proves difficult because transformations affect the entire input uniformly, hindering the disentanglement of features that change under transformation from those that remain invariant. While moving these operations to the latent space offers more flexibility, existing latent-space methods still rely on a predefined, manual partitioning of latent variables into equivariant and invariant subspaces. This dependency limits the robustness and scalability of the models.

To overcome these limitations, the authors propose a novel end-to-end framework designed to learn group actions on latent image manifolds by automatically discovering transformation-relevant structures. The core innovation lies in the use of learnable binary masks coupled with the Straight-Through Estimator (STE). These masks dynamically partition the latent representation into two distinct components: a transformation-sensitive subspace and an invariant subspace. By employing STE, the authors circumvent the non-differentiability of the binary masking process, allowing the entire system to be trained via backpropagation. This allows the network to learn which latent dimensions should respond to group actions and which should remain constant.

The proposed method is integrated into a unified optimization framework that jointly optimizes for both latent disentanglement and the learning of group transformation mappings. A significant advantage of this approach is its architectural agnosticism; it can be seamlessly integrated into any standard encoder-decoder architecture, making it highly versatile for various generative models. The effectiveness of the framework was rigorously validated across five diverse 2D and 3D image datasets. The results demonstrate that the model can autonomously identify and disentangle the latent factors necessary for group actions. Furthermore, downstream classification tasks were used to confirm that the learned representations are highly informative and preserve the essential semantic properties of the data. This work represents a significant step toward fully automated, controllable generative modeling.