Forecast Aware Deep Reinforcement Learning for Efficient Electricity Load Scheduling in Dairy Farms

Dairy farming is an energy intensive sector that relies heavily on grid electricity. With increasing renewable energy integration, sustainable energy management has become essential for reducing grid dependence and supporting the United Nations Sustainable Development Goal 7 on affordable and clean energy. However, the intermittent nature of renewables poses challenges in balancing supply and demand in real time. Intelligent load scheduling is therefore crucial to minimize operational costs while maintaining reliability. Reinforcement Learning has shown promise in improving energy efficiency and reducing costs. However, most RL-based scheduling methods assume complete knowledge of future prices or generation, which is unrealistic in dynamic environments. Moreover, standard PPO variants rely on fixed clipping or KL divergence thresholds, often leading to unstable training under variable tariffs. To address these challenges, this study proposes a Deep Reinforcement Learning framework for efficient load scheduling in dairy farms, focusing on battery storage and water heating under realistic operational constraints. The proposed Forecast Aware PPO incorporates short term forecasts of demand and renewable generation using hour of day and month based residual calibration, while the PID KL PPO variant employs a proportional integral derivative controller to regulate KL divergence for stable policy updates adaptively. Trained on real world dairy farm data, the method achieves up to 1% lower electricity cost than PPO, 4.8% than DQN, and 1.5% than SAC. For battery scheduling, PPO reduces grid imports by 13.1%, demonstrating scalability and effectiveness for sustainable energy management in modern dairy farming.

💡 Research Summary

This paper addresses the challenge of reducing electricity costs and grid dependence for dairy farms, which are highly energy‑intensive operations that increasingly rely on intermittent renewable generation and time‑varying electricity tariffs. While reinforcement‑learning (RL) approaches have shown promise for load scheduling, most existing methods assume perfect knowledge of future prices or generation, an unrealistic assumption in real‑world settings. Moreover, standard Proximal Policy Optimization (PPO) uses fixed clipping or KL‑divergence thresholds, which can cause unstable training when tariffs fluctuate sharply.

To overcome these limitations, the authors propose a novel deep‑RL framework that combines two key innovations: (1) Forecast‑Aware PPO (F‑PPO), which augments the state representation with short‑term forecasts of farm demand and renewable output. Forecasts are generated by a simple residual‑calibration technique that extracts hour‑of‑day and month seasonal components from historical data and predicts the remaining residual using a lightweight time‑series model. By providing the agent with a one‑hour‑ahead view of demand, generation, and price, the policy can anticipate upcoming peaks and valleys and act proactively. (2) PID‑KL PPO, a variant of PPO that replaces the static clipping or KL‑target with a proportional‑integral‑derivative (PID) controller that continuously adjusts the KL‑penalty based on the current KL divergence, the desired KL target, and the rate of change of KL. This adaptive mechanism stabilises policy updates under volatile price conditions, prevents excessive policy jumps, and accelerates convergence.

The architecture follows a standard actor‑critic design. The actor outputs discrete actions for the water‑heater (on/off) and continuous actions for battery charge/discharge power. The critic estimates the state‑value function. The reward function is a weighted sum of (i) electricity cost (time‑varying tariff), (ii) grid import/export cost (including revenue from selling stored energy), (iii) penalties for violating operational constraints such as battery state‑of‑charge limits, minimum heater runtime, and charging efficiency losses. Large penalties enforce realistic farm operating rules during training.

Experimental evaluation uses a real‑world dataset from an Irish dairy farm covering one year of hourly electricity consumption, on‑site solar and wind generation, and 15‑minute market prices. Missing values are linearly interpolated; seasonal residuals are removed using month‑wise averages; the resulting series is fed to the RL agents as a 24‑hour rolling window. Training proceeds over 30‑day episodes, repeated 2 000 times, with Adam optimizer (learning rate = 3e‑4), discount factor γ = 0.99, GAE λ = 0.95, and minibatch size = 64. Baselines include vanilla PPO, Deep Q‑Network (DQN), and Soft Actor‑Critic (SAC), all trained under identical conditions.

Performance is measured by (a) total electricity cost, (b) grid import volume, (c) battery cycle count and state‑of‑charge variance, and (d) policy stability (average KL divergence and its standard deviation). The proposed F‑PPO + PID‑KL achieves a 1 % cost reduction compared with vanilla PPO, 4.8 % versus DQN, and 1.5 % versus SAC. In the battery‑scheduling scenario, grid imports drop by 13.1 %, indicating better utilisation of on‑site renewables. Battery SOC fluctuations decrease by 8 %, suggesting longer battery life. KL divergence stays below the target of 0.02 (average ≈ 0.015) with a 30 % lower variance than standard PPO, confirming the stabilising effect of the PID controller. Training converges roughly 15 % faster than the baseline PPO.

The study also discusses limitations. The residual‑calibration forecaster is simple and may not capture complex weather‑driven generation spikes; more sophisticated LSTM or Transformer models could improve accuracy. The PID‑KL controller introduces three hyper‑parameters (Kp, Ki, Kd) that may need retuning for different farms or market regimes, potentially increasing deployment complexity. Future work is suggested to integrate advanced forecasting, multi‑agent coordination across multiple farms, and hardware‑efficient implementations for real‑time operation.

In conclusion, by embedding short‑term forecasts into the policy and adaptively regulating KL divergence with a PID controller, the paper delivers a robust, sample‑efficient RL solution that reduces electricity costs, lowers grid dependence, and respects realistic operational constraints in dairy‑farm energy management. The approach is readily extensible to other energy‑intensive sectors facing similar renewable‑integration and tariff‑volatility challenges.