Designing a Robust, Bounded, and Smooth Loss Function for Improved Supervised Learning

The loss function is crucial to machine learning, especially in supervised learning frameworks. It is a fundamental component that controls the behavior and general efficacy of learning algorithms. However, despite their widespread use, traditional loss functions have significant drawbacks when dealing with high-dimensional and outlier-sensitive datasets, which frequently results in reduced performance and slower convergence during training. In this work, we develop a robust, bounded, and smooth (RoBoS-NN) loss function to resolve the aforementioned hindrances. The generalization ability of the loss function has also been theoretically analyzed to rigorously justify its robustness. Moreover, we implement RoboS-NN loss in the framework of a neural network (NN) to forecast time series and present a new robust algorithm named $\mathcal{L}{\text{RoBoS}}$-NN. To assess the potential of $\mathcal{L}{\text{RoBoS}}$-NN, we conduct experiments on multiple real-world datasets. In addition, we infuse outliers into data sets to evaluate the performance of $\mathcal{L}{\text{RoBoS}}$-NN in more challenging scenarios. Numerical results show that $\mathcal{L}{\text{RoBoS}}$-NN outperforms the other benchmark models in terms of accuracy measures.

💡 Research Summary

The paper introduces a novel regression loss function called RoBoS‑NN (Robust, Bounded, and Smooth Neural Network loss) designed to address the well‑known sensitivity of traditional loss functions (MSE, MAE, Huber, Log‑cosh) to high‑dimensional data and outliers. The authors start by motivating the need for a loss that is simultaneously bounded (to prevent extreme outliers from dominating the objective), smooth (to retain differentiability everywhere), and robust (to down‑weight large residuals).

Loss formulation
For a residual (u = |y - \hat y|) the proposed loss is
\