A Hybrid Autoencoder for Robust Heightmap Generation from Fused Lidar and Depth Data for Humanoid Robot Locomotion

Reliable terrain perception is a critical prerequisite for the deployment of humanoid robots in unstructured, human-centric environments. While traditional systems often rely on manually engineered, single-sensor pipelines, this paper presents a learning-based framework that uses an intermediate, robot-centric heightmap representation. A hybrid Encoder-Decoder Structure (EDS) is introduced, utilizing a Convolutional Neural Network (CNN) for spatial feature extraction fused with a Gated Recurrent Unit (GRU) core for temporal consistency. The architecture integrates multimodal data from an Intel RealSense depth camera, a LIVOX MID-360 LiDAR processed via efficient spherical projection, and an onboard IMU. Quantitative results demonstrate that multimodal fusion improves reconstruction accuracy by 7.2% over depth-only and 9.9% over LiDAR-only configurations. Furthermore, the integration of a 3.2 s temporal context reduces mapping drift.

💡 Research Summary

The paper presents a learning‑based perception‑to‑control framework for humanoid robots operating in unstructured, human‑centric environments. The core idea is to generate a robot‑centric heightmap as an intermediate representation that feeds directly into a deep reinforcement‑learning (DRL) locomotion policy. To obtain this heightmap, the authors fuse data from an Intel RealSense depth camera and a LIVOX MID‑360 LiDAR, processing the LiDAR point cloud through an efficient spherical projection that yields a 276 × 40 range image. Both modalities are fed into separate convolutional neural network (CNN) encoders, each producing a 256‑dimensional latent vector. These vectors are concatenated with a 15‑dimensional robot state vector (derived from the IMU) and the heightmap predicted at the previous timestep (165 dimensions). The combined 448‑dimensional multimodal embedding is normalized and passed through two stacked Gated Recurrent Unit (GRU) layers (hidden size 256) that model a temporal window of 3.2 seconds (32 frames). A lightweight decoder head finally outputs a 165‑dimensional heightmap covering a 0.98 m × 0.7 m area with 7 cm cell resolution.

Training proceeds in two stages. First, symmetric autoencoders for each sensor modality are pretrained in an unsupervised fashion on noisy simulated data (Gaussian noise σ = 1 cm, random occlusion up to 3 %). Pixel‑wise mean‑squared error (MSE) loss is used, and skip connections are deliberately omitted to force all terrain‑relevant information through the latent bottleneck. In the second stage, the pretrained CNN encoders are frozen, the GRU temporal core is added, and the whole encoder‑decoder structure (EDS) is trained supervised on ground‑truth heightmaps generated in simulation. The dataset comprises 400 k samples per modality, split 70/15/15 for training/validation/testing, and is optimized with AdamW and a plateau learning‑rate schedule over 40 epochs. The final system runs at 10 Hz on embedded hardware.

Quantitative evaluation shows that multimodal fusion reduces mean absolute error (MAE) to 2.19 cm, a 7.2 % improvement over depth‑only (2.36 cm) and a 9.9 % improvement over LiDAR‑only (2.43 cm). A temporal context of 3.2 s yields the lowest reconstruction error; extending to 6.4 s provides diminishing returns. Accuracy remains below 2 cm on flat or gently varying terrain, but degrades on discontinuous surfaces such as stairs because the pixel‑wise MSE loss smooths sharp height changes into gradual slopes.

The reconstructed heightmaps are fed to a PPO‑trained locomotion policy in Isaac Lab. The optimized heightmap (size and resolution) enables anticipatory gait patterns: the robot lifts its swing leg in advance of upcoming elevation changes, reducing fall‑induced episode terminations by 70.1 % compared to a baseline without heightmap input. Command tracking improves, with linear velocity error reduced by 25 % and angular velocity error by 17 %. The policy remains robust when Gaussian noise up to 2 cm is added to the heightmap, but performance deteriorates sharply beyond this threshold, especially on terrain with abrupt height steps.

The authors discuss limitations: the 2.5‑D heightmap cannot faithfully represent vertical discontinuities, and the use of pixel‑wise MSE biases the network toward smooth global fits, compromising foothold precision on stairs. Moreover, the training data are predominantly simulated, leaving real‑world generalization unverified. Future work is suggested to incorporate multi‑scale loss functions, higher‑dimensional volumetric representations, and extensive real‑robot experiments to bridge the sim‑to‑real gap and better capture high‑frequency terrain features.