Modeling Cascaded Delay Feedback for Online Net Conversion Rate Prediction: Benchmark, Insights and Solutions

In industrial recommender systems, conversion rate (CVR) is widely used for traffic allocation, but it fails to fully reflect recommendation effectiveness because it ignores refund behavior. To better capture true user satisfaction and business value, net conversion rate (NetCVR), defined as the probability that a clicked item is purchased and not refunded, has been proposed.Unlike CVR, NetCVR prediction involves a more complex multi-stage cascaded delayed feedback process. The two cascaded delays from click to conversion and from conversion to refund have opposite effects, making traditional CVR modeling methods inapplicable. Moreover, the lack of open-source datasets and online continuous training schemes further hinders progress in this area.To address these challenges, we introduce CASCADE (Cascaded Sequences of Conversion and Delayed Refund), the first large-scale open dataset derived from the Taobao app for online continuous NetCVR prediction. Through an in-depth analysis of CASCADE, we identify three key insights: (1) NetCVR exhibits strong temporal dynamics, necessitating online continuous modeling; (2) cascaded modeling of CVR and refund rate outperforms direct NetCVR modeling; and (3) delay time, which correlates with both CVR and refund rate, is an important feature for NetCVR prediction.Based on these insights, we propose TESLA, a continuous NetCVR modeling framework featuring a CVR-refund-rate cascaded architecture, stage-wise debiasing, and a delay-time-aware ranking loss. Extensive experiments demonstrate that TESLA consistently outperforms state-of-the-art methods on CASCADE, achieving absolute improvements of 12.41 percent in RI-AUC and 14.94 percent in RI-PRAUC on NetCVR prediction. The code and dataset are publicly available at https://github.com/alimama-tech/NetCVR.

💡 Research Summary

This paper addresses the limitation of traditional post‑click conversion rate (CVR) prediction in industrial recommender systems, which neglects post‑purchase refunds and therefore fails to capture true user satisfaction and net business value. The authors introduce Net Conversion Rate (NetCVR) as the probability that a clicked item is both purchased and not refunded, explicitly modeling the two‑stage cascaded delay feedback: Click → Conversion and Conversion → Refund. Because the two delays have opposite effects—conversion delay positively contributes to the target while refund delay subtracts from it—standard CVR methods cannot be directly applied. Moreover, the field lacks publicly available datasets and online continuous training pipelines, hindering progress.

To fill this gap, the authors construct CASCADE, the first large‑scale open dataset dedicated to NetCVR prediction. CASCADE is derived from display‑advertising logs of the Taobao app and records the full user journey with precise timestamps for clicks, conversions, and refunds. This enables realistic streaming‑based experiments and supports the development of online continuous learning algorithms.

Through extensive exploratory analysis on CASCADE, three key insights emerge: (1) NetCVR exhibits strong intra‑day and inter‑day temporal dynamics, with notable drops during late‑night hours, indicating that daily‑batch updates are insufficient and that models must be updated continuously. (2) Modeling NetCVR as a cascade of two separate tasks—CVR prediction followed by Refund Rate (RFR) prediction—and then combining the outputs (NetCVR = CVR × (1 – RFR)) yields superior performance compared with directly predicting NetCVR. (3) The delay times between click‑conversion and conversion‑refund are informative features; shorter delays correlate with higher CVR and higher RFR, suggesting that delay time encodes user intent and satisfaction signals.

Guided by these insights, the paper proposes TESLA (Temporal‑aware Efficient Sequential Learning Architecture). TESLA comprises three main components: (i) a shared‑bottom multi‑task neural architecture that jointly learns CVR and RFR while allowing task‑specific towers to capture distinct patterns; (ii) stage‑wise debiasing mechanisms that correct for partially observed labels—using a short observation window for conversion (e.g., 3 days) and a longer window for refunds (e.g., 6 days)—to mitigate label‑delay bias; and (iii) a delay‑time‑aware ranking loss that assigns higher weight to samples with shorter observed delays, thereby encouraging the model to prioritize timely signals.

TESLA operates in an online continuous learning setting. Incoming events are processed in real time; conversion events are fed into the model immediately, while refund events are incorporated when they occur. Two observation windows dynamically update the ground‑truth labels, and the model is refreshed via mini‑batch stochastic gradient descent on each stream batch. This design allows TESLA to adapt to evolving user behavior and market conditions without the latency of daily retraining.

Empirical evaluation on the CASCADE benchmark compares TESLA against a range of baselines: traditional offline CVR models, delayed‑feedback methods such as DFM and NoDeF, and an adapted online continual learning method (FNC) for NetCVR. Metrics include RI‑AUC, RI‑PRAUC, negative log‑likelihood (NLL), and the post‑conversion‑over‑conversion (PCOC) ratio. TESLA consistently outperforms all baselines, achieving an absolute improvement of 12.41 % in RI‑AUC and 14.94 % in RI‑PRAUC over the strongest competitor. The cascade approach yields higher AUC and PR‑AUC than direct NetCVR modeling, confirming that decomposing the problem reduces bias and improves ranking quality. The delay‑aware loss further contributes a measurable gain, demonstrating the practical value of incorporating temporal signals.

Finally, the authors release the full codebase and the CASCADE dataset on GitHub, fostering reproducibility and encouraging future research on multi‑stage delayed feedback problems. The work not only advances NetCVR prediction but also provides a blueprint for other domains where outcomes unfold over multiple delayed stages, such as ad click‑install‑uninstall pipelines, subscription‑cancellation flows, or multi‑step fraud detection.