Foundations of the Theory of Performance-Based Ranking

Ranking entities such as algorithms, devices, methods, or models based on their performances, while accounting for application-specific preferences, is a challenge. To address this challenge, we establish the foundations of a universal theory for performance-based ranking. First, we introduce a rigorous framework built on top of both the probability and order theories. Our new framework encompasses the elements necessary to (1) manipulate performances as mathematical objects, (2) express which performances are worse than or equivalent to others, (3) model tasks through a variable called satisfaction, (4) consider properties of the evaluation, (5) define scores, and (6) specify application-specific preferences through a variable called importance. On top of this framework, we propose the first axiomatic definition of performance orderings and performance-based rankings. Then, we introduce a universal parametric family of scores, called ranking scores, that can be used to establish rankings satisfying our axioms, while considering application-specific preferences. Finally, we show, in the case of two-class classification, that the family of ranking scores encompasses well-known performance scores, including the accuracy, the true positive rate (recall, sensitivity), the true negative rate (specificity), the positive predictive value (precision), and F1. However, we also show that some other scores commonly used to compare classifiers are unsuitable to derive performance orderings satisfying the axioms.

💡 Research Summary

The paper tackles the fundamental problem of ranking algorithms, devices, methods, or models based on their performance while taking into account application‑specific preferences. The authors argue that current practice often conflates “performance” with ad‑hoc numerical scores and relies on intuition rather than a solid theoretical foundation. To remedy this, they construct a rigorous mathematical framework that unifies probability theory and order theory, and then build an axiomatic theory of performance‑based ranking on top of it.

Core components of the framework

1. Performance as a probability measure – Each entity’s performance is modeled as a probability measure (P) defined on a common measurable space ((\Omega,\Sigma)). This captures uncertainty about outcomes and allows different tasks (binary classification, regression, etc.) to be expressed by choosing an appropriate (\Omega).
2. Pre‑order (\preceq) on performances – A binary relation that is reflexive and transitive. From (\preceq) the authors derive the usual comparative notions: equivalence ((\sim)), strict superiority ((>)), strict inferiority ((<)), and incomparability ((\perp)).
3. Satisfaction variable (S:\Omega\rightarrow\mathbb{R}) – A task‑specific random variable that assigns a numeric “satisfaction” value to each elementary outcome. The expectation of (S) under a performance measure quantifies how good that performance is for the task.
4. Evaluation function (\mathrm{eval}:E\rightarrow\mathcal{P}(\Omega,\Sigma)) – Maps each entity (e) to its performance distribution. A higher‑level operator (\Phi:2^{\mathcal{P}}\rightarrow2^{\mathcal{P}}) captures knowledge about achievable performances (e.g., by combining or perturbing entities) and is idempotent.
5. Score functions (X) – Real‑valued functions that map a performance to a numeric score (accuracy, error rate, etc.).
6. Importance variable (I) – A random variable that encodes application‑specific preferences (e.g., higher weight on false negatives).

Axiomatic foundation
Three axioms are introduced:

• A1 (Compatibility) – The pre‑order must be consistent with the satisfaction variable (if one performance yields higher expected satisfaction, it should be ranked at least as good).
• A2 (Score monotonicity) – Any score used to induce a ranking must preserve the pre‑order.
• A3 (Importance integration) – When importance is taken into account, the induced ranking must still respect the pre‑order.

The authors prove three theorems giving sufficient conditions for a score and an importance distribution to satisfy all three axioms.

Ranking scores family
A parametric family of “ranking scores” is defined: \