Learning Memory-Enhanced Improvement Heuristics for Flexible Job Shop Scheduling

The rise of smart manufacturing under Industry 4.0 introduces mass customization and dynamic production, demanding more advanced and flexible scheduling techniques. The flexible job-shop scheduling problem (FJSP) has attracted significant attention due to its complex constraints and strong alignment with real-world production scenarios. Current deep reinforcement learning (DRL)-based approaches to FJSP predominantly employ constructive methods. While effective, they often fall short of reaching (near-)optimal solutions. In contrast, improvement-based methods iteratively explore the neighborhood of initial solutions and are more effective in approaching optimality. However, the flexible machine allocation in FJSP poses significant challenges to the application of this framework, including accurate state representation, effective policy learning, and efficient search strategies. To address these challenges, this paper proposes a Memory-enhanced Improvement Search framework with heterogeneous graph representation–MIStar. It employs a novel heterogeneous disjunctive graph that explicitly models the operation sequences on machines to accurately represent scheduling solutions. Moreover, a memoryenhanced heterogeneous graph neural network (MHGNN) is designed for feature extraction, leveraging historical trajectories to enhance the decision-making capability of the policy network. Finally, a parallel greedy search strategy is adopted to explore the solution space, enabling superior solutions with fewer iterations. Extensive experiments on synthetic data and public benchmarks demonstrate that MIStar significantly outperforms both traditional handcrafted improvement heuristics and state-of-the-art DRL-based constructive methods.

💡 Research Summary

The paper addresses the Flexible Job‑Shop Scheduling Problem (FJSP), a combinatorial optimization challenge that has become increasingly relevant in Industry 4.0 environments where mass customization and dynamic production demand both flexible machine assignment and precise operation sequencing. While recent deep reinforcement learning (DRL) approaches have shown promise, most of them are constructive: they build a schedule step‑by‑step by assigning operations to machines. Constructive methods suffer from incomplete state information because the partial schedule lacks crucial constraints such as current machine loads and work‑in‑progress data, which often leads to sub‑optimal makespans.

In contrast, improvement‑based methods start from a complete feasible schedule and iteratively refine it by exploring a local neighbourhood. This paradigm naturally captures all constraints, but applying it to FJSP is non‑trivial due to (i) the one‑to‑many relationship between operations and eligible machines, (ii) the need for an action space that can modify both sequencing and machine assignment, and (iii) the dramatically enlarged solution space that raises the risk of getting trapped in local optima.

To overcome these obstacles, the authors propose MIStar (Memory‑enhanced Improvement Search), a novel DRL‑driven improvement framework specifically designed for FJSP. The key components are:

1. Heterogeneous Disjunctive Graph Representation – Traditional disjunctive graphs model only operation nodes and undirected arcs that indicate that two operations share a machine, but they lack explicit machine nodes. MIStar augments the graph with machine vertices and directed hyper‑edges that connect a machine node to the ordered list of operations processed on it. The resulting directed heterogeneous graph (→H = (O, M, C, E)) simultaneously encodes job precedence (C), machine assignment, and processing order (E). This representation is lossless for a complete schedule and provides a rich substrate for graph neural networks.

2. Memory‑Enhanced Heterogeneous Graph Neural Network (MHGNN) – The policy network first runs a heterogeneous message‑passing GNN on →H to obtain operation embeddings and machine embeddings. In parallel, a memory module stores compact embeddings of previously visited schedules. When evaluating the current state, the top‑K most similar historical embeddings are retrieved and aggregated via a soft‑voting mechanism, producing a “historical context” vector. This vector is concatenated with the current graph embeddings, giving the policy a sense of past experience and helping it escape local minima.

3. Action Space Based on Nopt2 Neighborhood – The authors adopt the Nopt2 move, which selects a critical‑path operation, removes it from its current machine sequence, and reinserts it into the optimal insertion interval of an alternative compatible machine. This move simultaneously changes the machine assignment and the operation order, drastically reducing the neighbourhood size compared to naïve swap or insertion moves while still providing sufficient flexibility for FJSP.

4. Parallel Greedy Exploration – At each decision step, the policy outputs a probability distribution over candidate Nopt2 moves. The top‑N candidates are evaluated in parallel on a GPU, each yielding an immediate reward (makespan improvement) and a penalty based on similarity to previously visited solutions. The move with the highest net reward is executed, updating the schedule. This parallel greedy scheme yields high‑quality improvements with far fewer iterations than traditional exhaustive local‑search.

5. MDP Formulation and Reward Design – The state is the full schedule encoded by →H; the action is a specific Nopt2 move; the reward consists of a gain term (positive if the makespan improves) and a penalty term (proportional to similarity to stored solutions). Early in training the gain dominates, encouraging rapid makespan reduction; later, the penalty grows, promoting exploration and diversity when improvements plateau.

Experimental Evaluation – The authors test MIStar on synthetic instances ranging from 30 to 200 jobs and 10 to 30 machines, as well as on public benchmarks (e.g., FT06, FT10). Baselines include classic meta‑heuristics (Tabu Search, ILS, GA) and state‑of‑the‑art DRL constructive methods that use GNNs. Results show that MIStar consistently reduces makespan by 15–30 % relative to the best baselines, often achieving near‑optimal solutions with 2–3× fewer training episodes. Ablation studies confirm that removing the memory module degrades performance by about 8–12 %, highlighting the importance of historical context. Moreover, the parallel greedy search cuts total runtime to less than 40 % of that required by Tabu Search for comparable solution quality.

Conclusions and Future Work – MIStar demonstrates that a carefully designed heterogeneous graph representation, combined with memory‑augmented graph neural networks and an efficient Nopt2‑based action space, can bring DRL‑driven improvement heuristics to the challenging domain of FJSP. The framework is size‑agnostic, meaning the same policy can be applied to problems of varying scale without retraining. Future directions suggested include extending the approach to multi‑objective scheduling (e.g., energy consumption, tardiness), integrating online learning with real‑time shop floor data, and exploring more sophisticated memory mechanisms such as meta‑learning or continual learning to further enhance exploration in even larger industrial settings.