Reinforcement Learning for Durable Algorithmic Recourse

Algorithmic recourse seeks to provide individuals with actionable recommendations that increase their chances of receiving favorable outcomes from automated decision systems (e.g., loan approvals). While prior research has emphasized robustness to model updates, considerably less attention has been given to the temporal dynamics of recourse–particularly in competitive, resource-constrained settings where recommendations shape future applicant pools. In this work, we present a novel time-aware framework for algorithmic recourse, explicitly modeling how candidate populations adapt in response to recommendations. Additionally, we introduce a novel reinforcement learning (RL)-based recourse algorithm that captures the evolving dynamics of the environment to generate recommendations that are both feasible and valid. We design our recommendations to be durable, supporting validity over a predefined time horizon T. This durability allows individuals to confidently reapply after taking time to implement the suggested changes. Through extensive experiments in complex simulation environments, we show that our approach substantially outperforms existing baselines, offering a superior balance between feasibility and long-term validity. Together, these results underscore the importance of incorporating temporal and behavioral dynamics into the design of practical recourse systems.

💡 Research Summary

Algorithmic recourse aims to give individuals who are denied by automated decision systems actionable suggestions that would increase their chances of a favorable outcome upon re‑application. Existing work has largely focused on robustness to model updates or exogenous data shifts, while largely ignoring the endogenous feedback loops that arise when many applicants simultaneously act on the same recommendations, especially in competitive, resource‑constrained settings. In such environments, the decision threshold can shift as applicants improve their features, causing previously valid recourse to become invalid and eroding trust.

This paper introduces a time‑aware recourse framework that explicitly models how a population of candidates evolves over discrete time steps. At each step, a fixed number of new applicants enter, a limited number of slots are filled, and rejected candidates receive counterfactual feature vectors (X_CF) designed to guarantee acceptance within a predefined horizon T. The framework incorporates realistic behavioral aspects: each feature has a difficulty parameter d_i, a global difficulty β influences the probability of successful modification, dropout probability rises with repeated failures, and re‑application probability depends on confidence and urgency. These extensions go beyond prior simulations that assumed immediate re‑application, uniform difficulty, and zero dropout.

The authors cast the problem as a Partially Observable Markov Decision Process (POMDP). The latent state s_t contains the full configuration of the environment—features, identifiers, scores of all current and re‑applying candidates. The action a_t is a matrix of counterfactual vectors for every rejected applicant. Transition dynamics proceed in three phases: (1) accepted candidates permanently exit, (2) rejected candidates either drop out or modify their features according to the recommended changes (with stochastic success), and (3) a new application round occurs with fresh entrants and re‑applicants. Observations o_t consist only of the currently applying candidates, making the problem partially observable.

A novel multi‑objective reward function balances three desiderata. (i) Equity is promoted by minimizing the Gini index of the target scores g_t