Effective Online 3D Bin Packing with Lookahead Parcels Using Monte Carlo Tree Search

Online 3D Bin Packing (3D-BP) with robotic arms is crucial for reducing transportation and labor costs in modern logistics. While Deep Reinforcement Learning (DRL) has shown strong performance, it often fails to adapt to real-world short-term distribution shifts, which arise as different batches of goods arrive sequentially, causing performance drops. We argue that the short-term lookahead information available in modern logistics systems is key to mitigating this issue, especially during distribution shifts. We formulate online 3D-BP with lookahead parcels as a Model Predictive Control (MPC) problem and adapt the Monte Carlo Tree Search (MCTS) framework to solve it. Our framework employs a dynamic exploration prior that automatically balances a learned RL policy and a robust random policy based on the lookahead characteristics. Additionally, we design an auxiliary reward to penalize long-term spatial waste from individual placements. Extensive experiments on real-world datasets show that our method consistently outperforms state-of-the-art baselines, achieving over 10% gains under distributional shifts, 4% average improvement in online deployment, and up to more than 8% in the best case–demonstrating the effectiveness of our framework.

💡 Research Summary

The paper tackles the practical problem of online three‑dimensional bin packing (3D‑BP) in logistics centers where robotic arms must place parcels arriving on a conveyor belt one by one. While deep reinforcement learning (DRL) policies have recently outperformed classic heuristics, they are trained on a static pool of parcels and therefore suffer severe performance drops when short‑term distribution shifts occur—for example, when a new batch of parcels from a different warehouse arrives. Modern logistics infrastructure, however, provides a “look‑ahead” queue: as parcels pass a scanner their dimensions become known before they are placed, giving the system a short‑term forecast of the upcoming item distribution.

The authors formalize the problem as a Model Predictive Control (MPC) task with a finite horizon N. At each decision step t they seek the action sequence ((a_t,…,a_{t+N-1})) that maximizes the sum of immediate packing rewards (normalized packed volume) plus the value of the state after N steps, where the value is estimated by an offline‑trained critic (V_{\theta}). The discount factor is set to 1 because each episode ends when the bin is full. After solving the optimization, only the first action is executed and the process repeats with an updated look‑ahead queue, exactly the receding‑horizon principle of MPC.

A brute‑force search of the exponential tree (|A|^N) is infeasible, so the authors adapt Monte Carlo Tree Search (MCTS). Standard MCTS (e.g., the PUCT rule used in AlphaZero) relies on a policy prior (P_{\pi}(s,a)) that is assumed reliable across all states. In the presence of distribution shifts this assumption breaks: the prior may be misleading for rare look‑ahead patterns, causing the search to get stuck in sub‑optimal regions. To address this, the paper introduces a “Shift‑Aware” PUCT that interpolates between the learned prior and a uniform prior using a dynamic weight (\alpha(s)), called the Familiarity Score.

The Familiarity Score is computed offline by discretizing parcel dimensions into a finite vocabulary of M types and counting their frequencies in the training data. During search, for a node at depth k the algorithm looks at a fixed‑size local window (empirically set to three parcels) of the look‑ahead queue, maps each parcel to its type, and multiplies the corresponding type probabilities. This product yields (\alpha(s)\in