Cascaded Transfer: Learning Many Tasks under Budget Constraints

Many-Task Learning refers to the setting where a large number of related tasks need to be learned, the exact relationships between tasks are not known. We introduce the Cascaded Transfer Learning, a novel many-task transfer learning paradigm where information (e.g. model parameters) cascades hierarchically through tasks that are learned by individual models of the same class, while respecting given budget constraints. The cascade is organized as a rooted tree that specifies the order in which tasks are learned and refined. We design a cascaded transfer mechanism deployed over a minimum spanning tree structure that connects the tasks according to a suitable distance measure, and allocates the available training budget along its branches. Experiments on synthetic and real many-task settings show that the resulting method enables more accurate and cost effective adaptation across large task collections compared to alternative approaches.

💡 Research Summary

The paper addresses the challenge of learning a very large number of related tasks under strict resource constraints, a setting the authors refer to as Many‑Task Learning (MaTL). Traditional multi‑task learning (MTL) requires joint optimization of all tasks, which becomes infeasible when the number of tasks reaches the hundreds or thousands due to memory, communication, and computational overhead. Transfer learning (TL), on the other hand, treats each source‑target pair independently, scaling well but offering no systematic way to organize transfers across a massive task collection. To bridge this gap, the authors propose Cascaded Transfer Learning (CTL), a novel paradigm that propagates model parameters through a directed rooted tree connecting all tasks. The key steps are: (1) define a pairwise distance between tasks (e.g., based on parameter space, data distribution, or learned embeddings); (2) construct a complete weighted graph using these distances and extract a Minimum Spanning Tree (MST), which minimizes the total “transfer difficulty”; (3) select the medoid of the distance matrix as the root of the tree, ensuring a balanced and shallow hierarchy; (4) allocate a global training budget B across tasks, assigning each task a local budget b_v such that Σ b_v = B; (5) traverse the tree in topological order, initializing each child task with its parent’s refined parameters and performing a limited number of gradient‑based updates G_{b_v}. The authors assume that each refinement operator G_{b_v} is a contraction in parameter space with rate ρ_v < 1, a condition satisfied by gradient descent on strongly convex and smooth losses. Under this assumption, they prove that the error incurred at any node decays geometrically with the depth of the node, i.e., error ≤ ρ^{depth}·initial error. Consequently, a shallow MST rooted at the medoid yields the smallest possible error amplification. The theoretical contribution includes sufficient conditions under which a tree‑structured cascade outperforms both independent direct transfers (the “star” topology) and pairwise transfers without intermediate steps. The analysis also shows that the only structural requirement is that the tree respects the distance‑based ordering; any MST satisfies this when distances faithfully reflect transfer difficulty. Empirically, the authors evaluate CTL on synthetic data where task parameters are placed on a lattice, confirming that CTL reduces average loss by more than 15 % compared with direct transfers, especially under tight budgets. They further test CTL on large real‑world spatiotemporal forecasting benchmarks (SubseasonalClimateUSA and WeatherBench 2), involving thousands of localized climate prediction tasks. Using spatial proximity and representation similarity to define distances, CTL achieves 6–10 percentage‑point improvements in forecasting accuracy over a strong joint‑training baseline, while cutting total training time by over 30 %. An ablation study on budget allocation shows that weighting b_v by task difficulty (e.g., average distance to neighbors) yields additional gains in low‑budget regimes. The paper also discusses extensions such as multiple seeds (disjoint cascades), DAG structures allowing a task to receive information from several parents, and alternative graph constructions beyond MSTs. However, the core contribution remains the demonstration that a simple, distance‑driven MST cascade provides a scalable, theoretically justified, and empirically effective solution for many‑task learning under budget constraints. The work opens avenues for applying CTL in federated learning, distributed sensor networks, and personalized modeling where resources are limited and task relationships are only partially known.