Evaluating Phylogenetic Comparative Methods under Reticulate Evolutionary Scenarios

Phylogenetic comparative methods (PCMs) are widely used to study trait evolution. However, many evolutionary histories involve reticulate evolutionary scenarios, such as hybridization, that violate core assumptions of these methods. In this study, we evaluate how such violations affect the performance of PCMs. In particular, we focus on the ancestral character estimation, evolutionary rate estimation, and model selection. We simulate continuous trait evolution on various phylogenetic network topologies and assess the performance of PCMs that assume a bifurcating tree (i.e., major tree of the network) as the underlying model of evolution. We found that the performance of the tested PCMs was suboptimal. Using random forest, generalized linear models, and model-based clustering, we identified key factors contributing to these inaccuracies. Our results show that frequent and/or recent hybridization accompanied by one ore more transgressive events and rapidly evolving traits (i.e., high evolutionary rate) lead to significant estimation error, especially with respect to rate estimation and model choice. These factors substantially shift trait values away from tree-based model expectations, leading to overall increased error in parameter estimates. Our study demonstrates cases in which PCMs that rely on trees are likely to misinterpret biological histories and offers recommendations for researchers studying systems with complex evolutionary histories.

💡 Research Summary

This paper investigates how phylogenetic comparative methods (PCMs) that assume a bifurcating tree perform when the true evolutionary history follows a reticulate network, such as one shaped by hybridization or introgression. The authors simulate a large suite of species networks using the SiPhyNetworks R package, fixing speciation, extinction, and hybridization rates (1, 0.2, 0.25 respectively) and varying the proportion of lineage‑generative (LG) versus lineage‑neutral (LN) hybrid events. For each network they generate continuous traits under five evolutionary rate regimes (σ² = 0.01, 0.05, 1, 1.5, 2) and produce 100 independent traits per regime, yielding 500 trait datasets per network. All analyses are conducted on the “major tree” extracted from each network (the tree obtained by removing one incoming edge at each reticulation, keeping the edge with the highest inheritance probability γ).

Using standard tree‑based PCM pipelines, the authors estimate ancestral character states (ACE), the Brownian motion rate parameter (σ²), and select between Brownian Motion (BM) and Ornstein‑Uhlenbeck (OU) models via AIC/BIC. They then assess estimation error relative to the known true values from the network simulations. To identify which aspects of network topology and trait history drive error, they apply random‑forest importance ranking, generalized linear models (GLMs), and model‑based clustering.

Four key factors emerge as the strongest predictors of poor performance: (1) high frequency of hybridization events, especially when they occur recently; (2) the presence of transgressive evolution, where hybrid phenotypes lie outside the parental trait space; (3) high intrinsic evolutionary rates of the trait; and (4) asymmetric inheritance probabilities (γ far from 0.5). When these conditions co‑occur, tree‑based PCMs substantially misestimate ancestral states, over‑estimate σ², and frequently select the wrong evolutionary model (e.g., favoring OU when BM generated the data, or vice‑versa). The random‑forest analysis shows that hybridization frequency and recency together explain the largest proportion of variance in error, while GLMs confirm significant interaction effects between trait rate and transgressive events. Model‑based clustering reveals distinct error “regimes” that correspond to particular combinations of network topology (e.g., high LG proportion) and trait dynamics.

The authors conclude that relying on tree‑only PCMs in systems with documented reticulation can lead to misleading biological interpretations. They recommend (i) employing network‑aware comparative methods whenever possible; (ii) explicitly testing for recent or frequent hybridization before applying tree‑based analyses; (iii) considering transgressive phenotypic shifts as a source of model misspecification; and (iv) exercising caution with rapidly evolving traits, which are especially prone to bias. These guidelines aim to improve the robustness of comparative studies across the growing number of taxa where reticulate evolution is known or suspected.