Discrete stochastic processes, replicator and Fokker-Planck equations of coevolutionary dynamics in finite and infinite populations

ST A TISTICAL MODELS IN THE BIOLOGICAL SCIENCES BANA CH CENTER PUBLICA TIONS, VOLUME 80 INSTITUTE OF MA THEMA TICS POLISH ACADEMY OF SCIENCES W ARSZA W A 2008 DISCRETE STOCHASTIC PROCESSES, REPLICA TOR A ND F OKK ER-PLANCK EQUA TIONS OF COEVOLUTIONAR Y D YNAMICS IN FINI TE AND INFIN ITE P OPULA TIONS JENS CHRISTIAN CLAUSSEN Institute of The or etic al Physics and Astr ophysics, Christian-A l br e cht Uni versity Kiel L eibnizstr. 15, 24098 Kiel, Germany E-mail: cl aussen@the o-ph ysik.uni-kiel.de Abstract. Finite-size fluctuations in coevolutionary d y namics arise in mo dels of biologica l as w ell as of so cial and economic systems. This brief tutorial review surveys a systematic approac h starting from a stochastic process discrete b oth in time and state. The limit N → ∞ of an infinite p op u lation can b e considered exp licitly , generally leading to a replicator-t yp e eq uation in zero order, and to a F okker-Planc k -type equation in fi rst order in 1 / √ N . Consequences and relations t o some previous approaches are outlined. 1. In tro duction. Evolution is a biological pro c e ss ubiquituosly taking place, acting on on several temp or al, spatial, a nd taxonomic scales. B io logical organisms are co ded by their genetic sequence, and their a bilit y to s urvive, following Da rwin, is largely deter- mined b y their genes. How e ver, even the r elationship betw een genotype, phenotype and a fitness (e.g., defined by the r epro duction r ate) by no means is simple, as it maps an enormously high-dimensional space (the state space of a ll p oss ible genomes ) to a m uch low er -dimensional phenotype space, and finally to a one - dimensional space o f fitness v a l- ues. The extinction of species , as o bserved in the fossile rec ord, follows a co mplicated behaviour [Eld72] including intermittency-like long p erio ds o f stasis, the so- called “punc- tuated eq uilibria”. It is still at debate whether externa l or cosmic influences acco unt for these, or whether a purely dynamic mechanism of the evolutionary and ecolo gical dynam- ics can pro duce this st ylized fact. In this direction, the Bak-Snepp en model [Bak9 3] is a pioneering minimal mo del for the ex tinctio n of sp ecies, which how ever is difficult to rela te to the biological scenario . On the one hand, it provides an in termediate mo deling lev el, where biolog ical observ ations can b e explained to s ome extent, but on the other hand, 2000 M athematics Subje ct Classific ation : Primary 91A22; Secondary 60J20, 92D50, 92D15. Key wor ds and phr ases : evolutionary game theory , finite p opu lations, asymmetric conflicts. The pap er is in final form and no versio n of it will b e published elsewhere. [1] 2 JENS CHRISTIAN CLAUSSEN a mathematica l treatment is still p oss ible [Ba n05, Pis9 7]. On a n individua l- based level, the T angled Nature mo del [Jen04] approaches closer to biolog y . While the dyna mics at large will des erve some further deca des of resea r ch, simplified situations of co evolutionary dynamics, r estricting to a finite num b er of sp ecies can be studied by formal mo dels of evolutionary pro cesses , and can b e solved ana ly tically in many ca ses to a llow for a n e xem- plaric insight. In this brief tutor ial r e view, co evolutionary dynamics in finite po pulations is formulated within the framework of evolutionary game theory , providing a co n venien t common ma thematical framework for biolo gical, so cial and economical evolutionary pr o- cesses of strategies , o r genetic t yp e s. 1.1. Game t he ory: The str ate gy of c onflict. Game theo ry w as br ought in to play b y v on Neumann a nd Mo rgenstern [Neu53] as a minimal model to expla in and predict the b e- haviour of humans in strateg ic situations, b e them military or economic, assuming fully rational b ehaviour. In ga me theory , more sp ecific in t wo-pla y er normal form ga mes , agents 1 . . . N “play” a str ategy , o ut of a finite set of p ossible pure strateg ies. They interact with a par tner a ls o playing o ne of those strageg ies, and b oth receive a (real v alued) pay o ff ac- cording to a so- called pay off matrix. This is best illustr ated with a n exa mple. The most paradigma tic conflict situation describ ed in ga me theory is the Pris oner’s Dilemma. It is defined by the pay off matrix π ij =  3 0 5 1  . (1) This is to b e r ead as follows. The ga me is defined b etw een tw o play ers , who can adopt t wo po ssible strategies : to c o op e r ate (C) or to defect (D). The players act in para llel and ar e not informed ab o ut the oppo nent ’s mo ve. Each play er P (row player) playing strateg y i then receives the pay o ff π ij when meeting a player using strategy j (column player, O) as opp onent. Obviously , here the co nflict situation is symmetric, therefore the payoff matrix of the opp onent’s pay o ffs is given by the transp ose π T ij . Often bo th pay off matrices are combined to ( π P ij , π O ij ) =  3 , 3 0 , 5 5 , 0 1 , 1  . (2) In symmetric conflicts, thereby only r edundant information is added. In asymmetric con- flicts, for obvious reasons often called bimatrix g ames , the opp onent’s pay offs, in g eneral, can b e different. W e will analyze such cases in Section 6. In all ca ses ab ov e, the players play their res pective strategy with probability o ne, i.e., they play pur e strategies . A straightforward generaliz a tion is that play ers — in a random and uncorre lated manner — can use mixed stra tegies , or strategy profiles s n , i.e., p ossess a stra tegy vector (with co mpo nen ts ( s n ) i summing up to one, thus are elements o f a simplex S n ), and gain payoffs which are the corr espo nding linear c o m bination u n ( s n , s m ) = X ij ( s n ) i π ij ( s m ) j (3) for playing ag ainst one opp onent (pro file ) s n , a nd u n ( s n , { s m } ) = X ij m ( s n ) i π ij ( s m ) j (4) COEVOLUT IONAR Y DYNAMICS IN FINITE AND INFINITE POPULA TIONS 3 against a s et of o ppo nen ts indexed b y m . Now, can mixe d strategies exist that ar e suc- cessfull against all hyp o thetical sets of opp onents? This brings us to the central concept of a Nash e q uilibrium [Nas51]: A Nash equililbrium is a mixed strategy , in which no single agent c a n improv e its payoff by solely changing its own str ategy . 1.2. Nash e quilibria. A s trict Nash eq uililbr ium (resp., Nas h equililbrium) is defined as a strategy profile s ∗ = ( s ∗ 1 , . . . s ∗ N ) for whic h each agent’s strategy s n is a bes t (r esp., be s t or eq ual) resp onse to the strategies o f the other players s ∗ − n , i.e. ∀ n ∀ s ∈ S n ,s 6 = s ∗ n u n ( s ∗ n , s ∗ − n ) ≥ u n ( s, s ∗ − n ) (5) where > instead of ≥ mu st hold for a strict Nas h equilibr ium. Hereby s ∗ − n := ( s 1 , . . . s n − 1 , s n +1 , . . . s N ) is ca lled a strategy profile of the co -play er s (for- mally a set of N − 1 profiles, usua lly it is understo o d that playing against all co -play er s linearly sums the pay offs received playing with e a ch of them; so one can define the co- profile ( s ∗ − n ) k := ( N − 1) − 1 P i 6 = n ( s i ) k as a linea rly av eraged pro file of the co- play er s; or define u n ( ., s ∗ − n ) as the s um of pay offs ag ainst the profile of each co -play e r ). Nash’s theorem [Nas5 1] ensures that a normal- form game (as defined ab ov e) for a finite n um be r of strategies and a finite num b er of play ers alwa ys p ose s ses a Nash equi- librium. How ever, it ca n be degenerate or a mixed strategy (mixed Nash equlibrium) . In the ab ov e P risoner’s Dilemma, ((1 , 0) , (1 , 0)), i.e. b oth play e rs playing alwa ys “ defect”, is a Nash eq ulibr ium; hence tw o memory-less agents playing it for one round hav e no incen- tive to co op erate. Conv ersely , the Stag -Hunt g ame π ij =  5 0 3 3  has t w o symmetric; ((1 , 0 ) , (1 , 0)), and ((0 , 1) , (0 , 1)) the Hawk-Do ve g ame π ij =  ( V − C ) / 2 V 0 V / 2  with C > V has tw o nons y mmetric Nas h equilibria, ((1 , 0) , (0 , 1 )) and ((0 , 1) , (1 , 0)), the latter is an example of a po pulation where bo th stra tegies are present. 1.3. Evolutionary Game The ory and evolutionarily stable str ate gies. Decades after ga me theory w as inv en ted, Ma ynard Smith and P rice [May73] w ere the first to utilize its ap- proaches t o the under standing of biological conflict situa tio ns among whose the emergence of co op era tion [Axe 84] among anima ls and humans still is a co n tin uously active field. In the dy namical picture o f evolutionary ga me theory , the concept of the Nash equililbrium has its counterpart in the ev olutionarily stable strategies (ESS), which ar e defined as a p opulation in whic h a s ingle mutan t (c hanging to any of the p os sible strategies or genotypes) cannot inv ade the p opulation. In infinite p opulations, traditiona lly descr ibed within the framework of replicator equations [T ay78, Hof79, Zee80], ESS a ppea r as stable fixed po in ts (see [Hof98] for a sy stematic treatment). In finite p opulatio ns, how ever, this concept has to b e refined [No w04, T ay04, Nei04, Wil0 4, T r a06b]. F o r a mor e detailed int ro duction into the field of evolutionary ga me theory , a nd its recent developmen t, see the cla ssical textb o oks [Ho f84, Hof98] and recent rev iews [Sza07, Mie07]. 4 JENS CHRISTIAN CLAUSSEN 2. Microscopi c interact ions: Game theory based on particle collisio n mo de ls. The descr iption of so cia l agent b ehaviour as in teraction, or collision, o f particles has bee n studied by Helbing establishing a quite gener al framework [Hel92a, Hel92 b, Hel9 6], which a t that time had not further been tak en up within evolutionary game theo r y or evolutionary dynamics. One main limiting a ssumption of the mea n-field type desc r iption discussed in the next sections is tha t any spatial organiza tion can b e neg lected in a first order a pproximation. This approximation, as well as neglecting age structure a nd time delay [Alb04], howev er is not warranted in gener al. 2.1. Patchy e c osystems. T o describ e the full dynamics of eco logical and evolutionary pro cesses, it ca n b e neces s ary to split up the p opula tion in to parts, or patches, and to inv estiga te an in termediate level of sub divided po pulations [Che03] or metap opulatio n dynamics [H ˚ ar02]. In general, a rich v a riety of dynamical scenar ios can emer ge; thus simplified mo dels on lattices and graphs have b een investigated widely . 2.2. Sp atial mo dels. Life typically is organized, to a very rough approximation, as a c ov- ering o f the surface of earth, in competition for sunligh t, solid ground, or h un ting ter - ritory . So it is natural to inv e s tigate evolutionary dynamics of individuals lo cated in a tw o-dimensional space [Now92], where collective phenomena can emerg e [Her94]. An- nother classica l study by Lindg ren a nd Nor dahl [Lin94] inv estig ated the spa tial Pr isoner’s Dilemma g ame with strategies of differ en t memory le ngths. Spatial structure has be e n ident ified as one p o ssible mechanism to pro mote co op eration [Now92]. In genera l, spa- tial game theo ry is capa ble of rich dyna mica l behaviour [Sza9 9, Szo0 4, Sza05, T ra04], as coarsening , segr egation, and spiral w av es. Lik ewise, models for opinion dynamics hav e bee n studied, as the Szna jd model [Szn00]. 2.3. Evolutionary dynamics on gr aphs. The sy stematic understanding o f evolutionary dynamics o n graphs is a still de veloping field. An early investigation o f co evolutionary dynamics (i.e., including frequency-dep endence) on graphs ha s been given b y Eb el and Bornholdt [Ebe0 2], inv estigating the iterated Pr isoner’s Dilemma on netw orks. O n the small-world architecture, Szab´ o et al. [Sza04] inv es tigated the Ro ck-Paper-Scis s ors g ame dynamics. In [Lie05, Oht06 ], fixation prop erties of evolutionary dynamics on graphs are studied, and specia l subg raphs have b een identified to enhance or suppress fix ation. F or t wo strategies a nd no n frequency- de p endent fitness, Antal, Redner and So o d [Ant06] hav e provided exact r e sults for fixation for the cas e of degree-unco rrelated graphs. Despite this significa n t pr o gress [Sza07], a g eneral theory of co evolutionary dynamics on gra phs remains a formidable c hallenge. 2.4. Unstru ctur e d p opulation dynamics: Me anfield appr o ach, or P´ o lya urn mo dels. The approximation of an unstructured p opulation implies that individua ls are undistinguis h- able (apart fr o m their strategies or genotype), and individuals ar e chosen r a ndomly from the po pulation, for dea th, repr o duction, a nd co mpetition. Hence, the p opulation can be viewed as a P ´ oly a urn from w hich indiv idua ls are drawn for the sto ch astic pro cess. All evolutionary pro ces ses discuss ed for finite p opulations in the remainder b elong to this class o f pro cesses . COEVOLUT IONAR Y DYNAMICS IN FINITE AND INFINITE POPULA TIONS 5 3. Microscopi c ev o lutionary pro cesses Evolutionary pr o cesses hav e bee n widely considered in po pula tion genetics. F o r pro cesses in discrete time, an impo rtant systematic distinction has to b e made b etw een s ynchroneous or parallel [Baa97] update pro cesses, as the Fis her-W righ t pro cess, and pro cess e s with ov er lapping gener ations, whe r e one individual is replaced in ea c h evolutionary step (se e Fig. 1). 3.1. Fisher-Wright pr o c ess. The pro c ess defined b y Fisher [Fis30] and W right [W r i31] in its or iginal form is not freq uency-dep endent ; the fitness of the indiv idua ls do es no t dep end on the state of the p opulation (given by the num b er o f individuals in each of the strate- gies). The Fisher-W righ t pro cess is defined as follo ws. In each time s tep, all indiv idua ls repro duce with probabilities prop ortional to their fitness, unt il the same p opulation size N is reached. An impor tant case is given by neutral evolution , wher e all individuals have ident ical fitness. Hence, the discrete sto c hastic pro cess that describ es the time evolution resembles a rando m g enetic dr ift, and no Darwinian principle is incorp ora ted. ✐ ✐ ② ② ② ✐ ❄ ❆ ❆ ❆ ❯ ❆ ❆ ❆ ❯ ✁ ✁ ✁ ☛ ❄ ❄ ✐ ② ② ② ② ✐ ✐ ❄ ✐ ✐ ❄ ② ② ❄ ② ✐ ❄ ② ② ❄ ② ② ❄ ✐ Figure 1: Discr e te sto chastic evolution pro cesses: Fisher-W right pro cess (left) as a typi- cal synchroneous up date pro cess, and an ov erla pping generations pro cess (right) . In the Moran pro cess and Lo c a l up date the lifetime o f the individuals howev er is nonidentical, due to the sto chastic as ynchroneous update. F or the transition probabilities see text. The op en and clo sed bullets ca n represent tw o different strateg ies (in a so cial system), or t wo different allele s at a sp ecific genetic lo cus (in bio logy). The g eneralization to higher nu mbers o f stra tegies or genetic types is stra ightf orward. Of cours e the Fisher -W r ight pro cess can b e stra ight forwardly gener alized to the case where the fitness of strategies dep ends on pay off gains o btained fro m g a me-theoretic in- teractions with other individuals. The finite p opulation case has b een considered recently by T aylor and Now ak [T a y06]. Here the transition matrix of the Markov pro cess is by no means spa rse; apart from the absorbing b oundar ies all matrix elements ca n be nonzer o. 3.2. Discr ete sto chastic al pr o c esses for overlapping gener ations. The ev olutionary pro - cesses of Mor an or Lo cal upda te type pr ovide us with a transition probability T ± that the num ber of individuals i (being in the first s tr ategy) incr eases or decreases b y one, resp ectively . F o r definiteness, we consider explicitly the Mo ran evolution dynamics in arbitra ry 2 × 2 games [Cla05]. Given a finite p opulatio n of N age nts (tw o types/stra tegies A and B) 6 JENS CHRISTIAN CLAUSSEN int eracting via in a ga me with the pay off matrix P =  a b c d  . (6) In the frequency- de p endent Moran pr o c ess, Every ag e n t in teracts with a representativ e sample of the popula tion, yielding the pa yoffs of A and B indiv idua ls a s π A ( i ) = 1 − w + w a ( i − 1) + b ( N − i ) N − 1 (7) π B ( i ) = 1 − w + w c i + d ( N − 1 − i ) N − 1 , (8) i is the n um ber of A individuals, and 1 − w is a background fitness. 3.3. Mor an pr o c ess. In its original form, also the Mora n pro cess [Mor6 2] is not fre quency depe ndent. In this birth-de a th pro cess, in each time step an individual is selected for repro duction, and subsequently a ra ndomly selected indiv idua l die s . Hereby it is ensured that the total num be r of individuals N remains constant. F o r tw o strategies , the Mora n pro cess there fore is a o ne-dimensional Marko v pro ces s with a finite num ber of sta tes from i = 0 , . . . , N . T aking up the Moran approach, Now ak, T aylor, F uden b er g a nd Sasaki [Now04, T a y04] investigated a frequency- dependent Moran pro cess, defined as follows: In the frequency-dep endent Moran pro cess, selection for re- pro duction is pro po rtional to the payoff c ompared to the av er a ge pay off, π A ( i ) / h π i . The probability per time step that a copy of a n A agent is newb orn then is p + i/ N (with p + as given in T able 1 ). It repla c e s a randomly chosen individua l. Hereby , the fitness is ev a lua ted after each individua l to ok in teraction according to a pay off matr ix with all individuals in the populatio n. 3.4. L o c al u p date and imitation pr o c esses. An a pparent limitatio n of the Mor an pro ces s with resp ect to biolog ical situatio ns is that individuals in e a ch up date step have to com- pete with the whole p opula tio n (or , in rea l s y stems with a repr e sent ative sample) of the po pulation. A mor e r ealistic setting is given by lo ca l o r pairw is e co mpetetive interactions, w hich can b e called imitatio n dynamics [Hof00] (in the concept of so cial strategies ) or lo cal up- date. These pro cess es c a n be defined in a slightly differ e n t manner. In a non-sy mmetric definition of the (linear) lo cal up date pr o cess [T ra05], one individual is s elected for r e- pro duction, and the other for death. Then the str ategy of the r epro ducing individual is changed (or kept ) with probabilities that dep end linearly on the difference of the payoffs of the tw o in teracting individuals. Her eby , as is us ua lly ass umed in ev olutionary g ame theory , the payoffs ar e ev alua ted by pla ying (on a shorter time scale) with the whole po pulation, so these pa yoff differences still are ev aluated in a globa l pr o cess. 3.5. Nonline ar r esp onse: L o c al F ermi pr o c ess. A v aria n t [Blu9 3, Sza98, Hau05] o f the Lo cal update pr o cess a rises naturally when the game in ter action pay off is one of man y additive contributions to the repro ductive fitness. In the case wher e the external contribu- tions ar e la rge, and co nsist of many degrees o f freedom that act like an ex ter nal heatbath, the pay o ffs of the individuals app ear as arg umen t o f B o ltzmann facto rs, where a par ame- ter w can b e introduced a s for the other pro cesses ( w → 0 cor r esp onds to weak selection), COEVOLUT IONAR Y DYNAMICS IN FINITE AND INFINITE POPULA TIONS 7 T a ble 1: Compa rison of three microsco pic upda te pr o cesses and their resp ective differen- tial e quations for x := i/ N obtained in the deter ministic limit of N → ∞ (see Sec . 5 .2). Moran pro ces s Lo cal up date F er mi pro c e ss p + = 1 − w + w π A i 1 − w + w h π i i p + = 1 2 + w 2 π A i − π B i ∆ π max p + = 1 1 + e − w ( π A i − π B i ) p − = 1 − w + w π B i 1 − w + w h π i i p − = 1 2 + w 2 π B i − π A i ∆ π max p − = 1 1 + e − w ( π B i − π A i ) 0 ≤ w ≤ 1 0 ≤ w ≤ 1 0 ≤ w ≤ ∞ T r ansition probabilities for incr e a se/decrea se of i : T ± = p ± i N N − i N T ± = p ± i N N − i N T ± = p ± i N N − i N In the N → ∞ limit: ˙ x = wx (1 − x )( π A x − π B x ) 1 − w + w h π ( x ) i ˙ x = wx (1 − x )( π A x − π B x ) ∆ π max ˙ x = x ( 1 − x ) tanh( w 2 ( π A x − π A x )) adjusted r eplicator eq. ordinary r eplicator eq. replicator eq . after [T ra06c] and her e can b e interpreted as a temper a ture. T his pair wise compariso n pro cess has b een studied in detail in [T ra06c, T ra07]. See T able 1 for the transition pr o babilities T ± of the Moran pro ces s, Lo cal update, and F ermi pr o cess. 4. Broadening of di stributions in finite p opulations. F or the Moran pro cess , the strategy distr ibutio n is generated only by the inherent sto chasticit y o f the finite po pula- tion. A t the bo rders, for T 0 → 1 and T N → N − 1 we a ssume a sma ll mutation rate µ [F ud04]. While in the infinite p opulation ca se the p opulation density is p eaked at the determinis- tic tra jectory in the sense of a delta distribution, for a finite p opulation it is broadened. F o rtunately , the Mora n pro cess allows for a n analytical trea tmen t. The stationary dis tri- bution for an a rbitrary payoff matrix can b e expr e s sed v ia Po chhammer sy mbo ls, r ising factorials or gamma functions, for the general case of 2 × 2 games including nonv anishing background fitness [Cla0 5]. F or illustration, let us consider the specia l cases of neutral evolution, constant fitness , an ‘an tico ordination ga me’ and the Pris oner’s dilemma, P n =  a a a a  , P c =  a a c c  , P AC =  0 1 1 0  , P PD =  3 0 5 1  . Internal (mixe d) vs. external (pur e) Nash e quilibrium. F or frequency dep e nden t fitness and w > 0, the game can hav e an in ter nal Nash equilibr ium or an equilibrium in one of the absorbing states. A simple exa mple with an internal Nash e q uilibrium is the ‘anticoordina tion’ game with w = 1, P =  0 1 1 0  . The transition matrix here is T i → i +1 = N − i 2 N , T i → i − 1 = i 2 N , des c ribing a ra ndo m w alk with a drift towards the de- terministic fixed po int i = N / 2 . In equilibrium, we hav e P i T i → i +1 = P i +1 T i +1 → i for 8 JENS CHRISTIAN CLAUSSEN every i , thus P i +1 = P 0 i Y j =0 N − j j + 1 = P 0  N i + 1  . (9) P i is a binomial dis tribution aro und the equilibrium o f the r eplicator dynamics . The Pr isoner’s Dilemma has a Nash equilibrium for m utual defection, i.e . i = 0. At the bo rders, for T 0 → 1 and T N → N − 1 we assume a small mut ation ra te µ [F ud04]. As b = 0, also state i = 1 is a bs orbing for w = 1 (tw o co op er a tors are needed to promote co op era tion). Thu s a small mutation r ate µ ha s to b e assumed also for T 1 → 2 . Alternatively o ne co uld assume w < 1. T a ble 2 : The fo ur pay off matrix cases considered in Fig. 2 under the assumption of a small m utation rate (see text). Without the frequency-dep endence introduced b y a game, the cases of constant fitness and neutral evolution kno wn from mathematical genetics are recov ered; the distribution keeps lo ca lized (for low mu tation r ates, mutan ts sta y rare ). F o r the a nalytical expressions of the distributions s ee [Cla05]. The distributions follow approximately a stretched exp onential with the fit e x po nen ts γ a s given in the table. Pa yoff matr ix Distr ibutio n ≈ exp( − bx γ ) Nash equilibrium (NE)? a = b = c = d Neutral evolution P i ∼ 1 i ( N − i ) drift → i = 0 and → i = N a = b < c = d constant fitness ≃ exp onential γ = 0 . 87 drift → i = 0 P AC =  0 1 1 0  ≃ binomial γ = 2 . 07 i = N / 2 “internal NE” P P D =  3 0 5 1  γ = 0 . 63 i = 0 “external NE” The four ca ses are summar ized in T able 2. F or neutral e volution the deca y is very slow. With constant fitness , the probability dec ays approximately exp onentially . F or the Prisoner ’s dilemma, the decay b ecomes slower with larger distance, while it b ecomes fa ster for the bino mial distributio n at the internal Nash equilibrium. The distributions are shown in Fig. 2. The decay of the distribution can b e fitted by a P ≈ exp( − bx γ ), γ AC = 2 . 0 6, γ C F = 0 . 87, γ P D = 0 . 6 3 . This cor resp onds to random mo tion in a n a nharmonic p otential. 5. F rom microscopic equations to m acroscopic equations Corresp onding to the discreteness of states, we fir st des crib e the pro cesses by a master equation, then for m ulate a F okker-Planck (Kolmogor ov forward) equation for large N , and finally compare the deterministic equations r e sulting from the limit N → ∞ . 5.1. Limit of lar ge p opulations: F r om Master e quation to F okk er-Planck e qu ation. The equation of motion (or time evolution equa tion) for the s to chastic pr o cess c a n b e for m u- COEVOLUT IONAR Y DYNAMICS IN FINITE AND INFINITE POPULA TIONS 9 10 -5 10 -4 10 -3 10 -2 10 -1 10 0 0 10 20 30 40 50 P j /P max j-j max Neutral evolution, w=0 Anticoordination game Constant fitness r=0.5 Prisoner’s Dilemma -Log(P j /P max ) j-j max Figure 2: Repro duced from [Cla05]. The inv ariant distr ibutions P j share different decay tails as a function of the distance j − j max from the Nash equilibrium j max . Her e r = 1 − w + w a . In the fo ur cases considered her e (shown for N = 100), the distributions can b e approximated by s tr etch ed exp onentials (which would a ppea r as straight lines in appropria te scaling, the inset shows that the approximation is reaso na ble). F or hig her m utation ra tes see [T ra 06a]. lated in terms of the master equa tion P τ +1 ( i ) − P τ ( i ) = P τ ( i − 1) T + ( i − 1) − P τ ( i ) T − ( i ) (10) + P τ ( i + 1) T − ( i + 1) − P τ ( i ) T + ( i ) for the probability distr ibution, i.e., P τ ( i ) is the pro bability to be in state i at time τ . F or N ≫ 1 w e c a n pr o ceed v ia a Kra mers-Moy al expansio n, defining x = i/ N and t = τ / N . Then a formal T a ylor expansion of T and ρ ( x, t ) = N P τ ( i ) yields, considering only the t wo lea ding terms of the T a y lor e x pansion, d dt ρ ( x, t ) = − d dx [ a ( x ) ρ ( x, t )] + 1 2 d 2 dx 2  b 2 ( x ) ρ ( x, t )  (11) a ( x ) = T + ( x ) − T − ( x ) (12) b ( x ) = r 1 N [ T + ( x ) + T − ( x )] . (13) F o r large, but finite N , this equation has the form of a F okker-Planck equa tion. This allows to g e ne r alize the diffusion approximation (see e.g. [Dro0 1]) to co evolution. Since the in ternal noise is not corre lated in time as subseq ue nt up date steps ar e indep endent, according to the Itˆ o calculus a corres po nding Langevin equation reads ˙ x = a ( x ) + b ( x ) ξ (14) where ξ is uncorr e lated Gaussian noise and b ( x ) = 0 for x = 0 and x = 1. W e here see that the noise is multiplicativ e and fr equency-dep endent. 10 JENS CHRISTIAN CLAUSSEN 5.2. Limit of infinite p opulations: R eplic ator e qu ation and A djuste d r eplic ator e quation. The leading or der term do es not v a nis h and describ es the deter minis tic drift term. Surprisingly , the replicator equatio n [T a y78, Hof79, Zee8 1] ˙ x = wx (1 − x )( π A x − π B x ) ∆ π max (15) is obtained for the Lo cal up date pro cess [T ra05], wher eas the adjusted replicato r equa tion ˙ x = wx (1 − x )( π A x − π B x ) 1 − w + w h π ( x ) i (16) is the deterministic limit of the Moran pro cess [T ra05]. F or other pro cesses, other differ- ent ial equa tions may b e o btained, see T able 1. The difference b etw e en the tw o equatio ns may b e b est illustrated for the Pr isoner’s Dilemma. Her e the (ordinary) replic a tor dynamics reads ˙ x = − x (1 − x 2 ), whereas the adjusted replicator dynamics reads ˙ x = − x (1 − x 2 ) 1 − w w +1+3 x − x 2 . In this case, the sta bilit y of fix ed po in ts is preser ved, as the a dditional denominator can b e abso r be d into a dynamical rescaling of the time sca le, commonly known as velocity transform. How ever, in a sym- metric conflicts the ch ange of time s c a le matters, as will b e deta iled in the next section. 6. Asymme tri c and cyclic g ames. Cyclicity of evolutionary dy namics intuit ively is contradictory to the ov ersimplified picture of a n abso lute fitness landscap e : any sp ecies having r eached the maximal fitness v alue within a po pulation will outcomp ete all others, and no cyclicity or oscilla tion can emerge. How e ver, in ecologica l comp etition, o scillations of p o pulations are quite generic, as known since Lotk a [Lot2 0] and V olterra [V ol26] (see also [Mur0 4]). Of sp ecial in terest a r e cases wher e sp ecies cy clically outcomp ete each other, as in Dawkins’ c a ricature of mating strategies (see Sections 6.1 – 6.2), or the children’s game Ro ck-Paper -Scissors , w her e ro ck crushes scis s ors, pap er covers ro ck, and scissors cuts pap er. This situation has b een sp otted in the ter ritorial b ehaviour of lizards [Sin96, Zam00], a nd a lso in E .coli bac ter ia in vitro [Ker02] and in vivo [Kir04]. The F okk er-Planck equation, in a na logy to Sec. 5.1, for the Ro ck-Paper -Scissors ga me has rec e n tly b een given in [Rei06], and generaliza tions to other cyclic evolutionary games a re straightforward. It is generally c laimed tha t such cy clic c o e volution pr omotes bio diversity [Csa02, Cla 08]. How ever, its relev ance in a more g eneral picture of “evolution at lar g e” remains a n is sue still to b e inv estigated. 6.1. Asymmetric c onflicts (bimatrix games). As mentioned in the intro duction, the pay- off matrix for the oppo nen t ca n b e different from that ea rned by the fir st player. T o set this scenario into work, it is usually required tha t the g ame is play e d b etw een t wo dis junct po pulations (as below, female and male), or that the in teraction pro c ess itself is asym- metric, e.g., that one play er is “ initiative” and gains pay o ffs different fro m those ear ne d in the opp onents’ role. Many so cial and ec o nomic situations b ear such asymmetries , but they a re often too weak to b e significantly ex tracted fro m data. An illustrative example of biolo g ical mating b ehaviour has been given by Dawkins [Daw76]. Male and female each can o cc ur with tw o genetic s trategies, “fast” (male: phi- landerer, female: “fast”) and “s low” (male: “fa ithful”, female: “coy”). The payoff bene fit COEVOLUT IONAR Y DYNAMICS IN FINITE AND INFINITE POPULA TIONS 11 of a child is assumed to b e b for b oth parents, the total cost of ra is ing an offspring is ( − 2 c ), and the prolo nged c ourtship that coy females insis ts on add a burden of a to b oth parents. Coy females and male philanderer s pro duce no thing a nd g ain nothing . The cost ( − 2 c ) is c overed b y b oth par ents, e xcept for philander er males and fast females, where the fema le has to g rowup its offspring alone. This tra nslates into the payoff ma trix (with usual par ameter choices a = 3 , b = 1 5 , c = 10, see also Fig. 3) ( π M ij , π F ij ) =  ( b − c − a, b − c − a ) ( b − c, b − c ) (0 , 0) ( b, b − 2 c )  =  (2 , 2) (5 , 5) (0 , 0) (15 , − 5)  . (17) If the four conditions a > 0 , c > 0 , b − c > 0 , 2 c − a − b > 0 are fulfilled, the game is cyclic. These c o nditions corresp ond to the ar rows in Fig. 3. The c yclicity of the g ame is preserved if we co nsider a normalized v ersion (being equiv alent to the pay o ff matrix of “Matching Pennies” played by tw o players): ( π M ij , π F ij ) =  (+1 , − 1) ( − 1 , +1) ( − 1 , +1) (+1 , − 1)  . (18) F o r the cyclicity refer to Fig. 3. Let the cycle start in the low er right corner , where fast females and male philandrer s are present. If males ar e philander ers, it pays for females to be coy (low er left corner ); insisting on a long cour tship per io d to make ma le s inv est more in the offspring (upp er left cor ner). How ever, once most males a r e faithful, fast females are fav or ed av oiding the costs of courtship (upper right corner ). Subsequently , the male inv estment in to the o ffspring is no longer justified, philanderer s are ag a in fa vored (low e r right co rner), and the cy c le contin ues . Philanderer F a ithful Male Coy F a st F ema le ✲ ✛ ❄ ✻ 0 0 15 − 5 2 2 5 5 Philanderer F a ithful Male Coy F a st F ema le ✲ ✛ ❄ ✻ − 1 +1 +1 − 1 +1 − 1 − 1 +1 Figure 3: Asymmetric pay o ff matrices for the B attle o f the Sexes [Da w76]. L e ft: Original pay o ff matrix suggested b y Dawkins. Right: Simplified pay off matrix, iden tical to the game “Matching p ennies” . In b oth cases, the left ent ry is for the row player, and the right ent ry (upo n a common c o nv ention, shifted upw a rds, to enha nc e intuitiv e a ssignment) is for the column play er. Arr ows indicate the cyclic dominance. This game e x hibits neutrally stable p er io dic orbits [May87] when descr ibed by the usual replic a tor equation approach [T a y78, Hof79, Zee81]. This would, acco rding to Dawkins, lead to an infinitely la sting oscillation of str a tegies. As Dawkins ar gues, certain sp ecies (gibbon, stickleback, duck, fruit fly) ha ve, in the course of r eal evolution, chosen to follow a fixed pa ir o f strategie s ; for them the “Ba ttle of the Sexes ” has come to a res t. Of course this mo dels relies on several assumptions, and in the r emainder we will analyze how the co nclusions change when co nsidering differ e n t evolutionary pro cesses and when explicitly considering the finiteness of the p opula tion. 12 JENS CHRISTIAN CLAUSSEN -0.002 -0.001 0.000 0.001 0.002 0.003 0.004 0.005 0 100 200 300 400 500 < ∆ H > N Population size N (a) 10 -6 10 -5 10 -4 10 -3 10 -2 10 1 10 2 10 3 10 4 10 5 10 1 10 2 10 3 10 4 0.0 0.1 0.2 0.3 0.4 0.5 N c Selection pressure w (b) Figure 4 : Repro duced fr o m [T ra05]. Drift reversal in the asymmetric ga me “Battle of the Sexes”. Shown is the av e r age change of H (which is a co nstant of motion in the ordinary replicator equation) for different p opulation sizes. (a) F or the Moran pr o cess (circles) above a critica l p opulation size the average change o f H b ecomes negative so that tr a jectories s pir al outw ards on av era ge. F or the Lo cal up date, no drift reversal is observed and for N → ∞ the change of H appr o aches zero (shown in the inset in double logarithmic plot). (b): Scaling o f the cr itical p opulation size with selection pressure w . 6.2. Counterintuitive b ehaviour for the “ Battle of the Sexes”. The case of asymmetric conflicts op ens the p ossibility of a co n terintuitiv e finite-size effect: The sign of the av er- age of H = − xy (1 − x )(1 − y ) (b eing a constant of motion for the ordina ry replicato r equation) c hanges for the Moran pro cess a bove a critical po pulation size, whereas the change remains p ositive (spiraling outw ar ds) for the Loca l up date and F ermi pr o cess in all finite p opulations (Fig. 4) [T ra05, Cla 06, Cla 07]. Hence in the N → ∞ limit the Moran pro cess s hows a deterministic b ehaviour (that o f the adjusted replica tor equa tion), which is qualitatively different from that of the ordinar y r eplicator equation. The r eason is that the co mmo n velocity tra nsformation, which can a bs orb the no rmalization denominato r of the av erage fitness, her e cannot b e p erformed, a s female and male p opulation ea rn differ- ent payoffs and the deno mina tors no longer coincide. The adjusted r eplicator equa tions here re ad ˙ x = +2 ( x 2 − x )(2 y − 1) 1 − w w + (2 y − 1)(2 x − 1) (19) ˙ y = − 2 ( y 2 − y )(2 x − 1) 1 − w w − (2 y − 1)(2 x − 1) . (20) Hence, if the tw o p opulations earn different average payoffs (here they differ b y a factor − 1), the fixed po int stability of b oth types of r eplicator equatio ns can differ. COEVOLUT IONAR Y DYNAMICS IN FINITE AND INFINITE POPULA TIONS 13 References [Alb04] J. Alboszta and J. Mi¸ ekisz, Stabilit y of evo lutionarily stable strategies in discrete replicator dynamics with time delay , J. Theor. Biol. 231 (2004), 175-179. [Ant06] T. Antal, S. Redner, and V. So od, Evo lutionary Dynamics on Degree-Heterogeneous Graphs , Phys. Rev. Lett. 96 (2006), 188104. [Axe84] R. Ax elrod, The evolution of coop eration , Basic Bo oks, New Y ork (1984). [Baa97] E. Baak e, M. Baak e, and H. W agner, Ising Quantum Chain is Equiv alen t to a Model of Biological Evolution , Phys. Rev. Lett. 78 (1997), 559-562. [Bak93] P . Bak and K. Snepp en, Pun ctuated equilibrium and criticali ty in a simple model of evolution , Phys. R ev. Lett. 71 (1993), 4083-4086. [Ban05] C. Bandt, The Discrete Evolution Mo d el of Bak and Snepp en is Conjugate to the Classica l Con tact Pro cess , J. Stat. Phys. 120 ( 2005), 685-693. [Blu93] L. E. Blume, The Statistical Mec hanics of Strategic Interaction , Games Econom. Beha v. 5 (1993), 387-424. [Che03] J. L. Cherry and J. W akely , A diffusion appro ximation for selection and drift in a sub divided p opulation , Genetics 163 (2003), 421-428. [Cla05] J. C. Claussen and A. T raulsen, Nongaussian fl uctuations arisi ng from fi nite p opulations: Ex act results for the evolutionary Moran pro cess , Ph ys. R ev. E 71 (2005), 025101(R) . [Cla06] J. C. Claussen and A. T raulsen, Fluctuations in Coevolutionary Dynamics and Im- plications for Multi-A gent Models , p. 411-419, in: D. Helbing (Ed .), Proc. P otentia ls of Complexity Science for Business, Gov ernment, and the Media, Budap est ( 2006). [Cla07] J. C. Cla ussen, D rift reversa l in asymmetric coevolutionary conflicts: I nfluence of microscopic p rocesses an d p opulation size , Eur. Phys. J. B 60 , 391-399 (2007). [Cla08] J. C. Claussen and A. T raulsen, Cyclic Dominance an d Biodiversit y in W ell-Mixed P opulations , Phys. Rev. Lett. 100 , (2008) 058104. [Csa02] T. L. Cz´ ar´ an, R. F. Ho ekstra and L. P agie, Chemical warfare b et w een microbes promotes b io diversit y , Pro c. Natl. Acad. Sci. USA 99 (2002), 786-790. [Da w76] R. Dawkins, The Selfish Gene , Oxford U niversi ty Press, N ew Y ork (1976). [Dro01] B. Dross el, Biolog ical evolution and statistical physics , Adv. Ph y s. 50 (2001), 209-295. [Ebe02] H. Eb el and S . Bornholdt, Co evolutionary games on netw orks , Phys. Rev. E 66 (2002), 056118 . [Eld72] N. Eldredge and S. J. Gould, Punctuated equilibria: an alternative to phyletic gradualism , in: T.J.M . Schopf, Ed itor, Mo dels in P aleobiolo gy , F reeman Co oper, San F rancisco, pp. 82-115 (1972). [Fis30] R. A. Fisher, The Genetical Theory of Natural Selection (Clarendon, Oxford, 1930). [F ud04] D . F udenberg and L. A. Imhof, Imitation Pro cesses with Small Mutations , Harv ard Institute of Economic Research Discussion Paper No. 2050 (2004). [H ˚ ar02] K. C. Harding and J. M. McNamara, A Unifying F ramew ork for Metap opulation Dynamics , Amer. N aturalist, 160 (2002), 173-185. [Hau05] C. Hauert and G. Szab´ o, Game theory and physics , Am. J. Phys. 73 (2005), 405-414. [Hel92a] D. H elbing, A Mathematical Mo del for Attitude F ormation b y P air Interactions , Beha vioral Science 37 (1992), 190-214. [Hel92b] D. Helbing, Phys. A 181(19 92), 29-52; Phys. A 193 (1993), 241-258 ; Ph ys. A 196 (1993), 546-573. [Hel96] D. H elbing, A stochastic behavioral mo del and a “microscopic” found ation of evol utionary game th eory , Theory and Decision 40 (1996), 149-179. 14 JENS CHRISTIAN CLAUSSEN [Her94] A. V. M. Herz, Collective Phen omena in Spatially Exten ded Ev olutionary Games , Journal of Theoretical Biology 169 (1994), 65-87. [Hof79] J. Hofbauer, P . Sch uster, and K. Sigmund, A note on evolutionary stable strategies and game d y namics , J. Theor. Biol. 81 (1979), 609-612. [Hof84] J. Hofbauer and K. Sigm und, Evo lutionstheorie und dynamisc he Systeme , Paul P arey , Berlin (1984). [Hof98] J. Hofbauer and K. Sigmund 1998, Ev olutionary Games and P opulation Dynamics (Cam bridge: Cam bridge Universit y Press) [Hof00] J. Hofbauer, K. H . Schlag, Sophisticated imitation in cy clic games , J. Ev ol. Econ. 10 (2000), 523-543 . [Jen04] H . J. Jensen, Emergence of sp ecies and pun ctuated equilibrium in the T angle Nature mo del of biological evolution , Phys. A 340 ( 2004), 697-704. [Ker02] B. Kerr, M. A. R iley , Marcus W. F eldman, and Brendan J. M. Bohannan, Lo cal disp ersal promotes bio diversit y in a real-life game of ro ck-paper-scissors , Nature 418 (2002), 171-174. [Kir04] B. C. Kirkup and M. A. Riley , Antibiotic-mediated antagonism leads t o a b acterial game of ro ck-paper- scissors in vivo , Nature 428 (2004), 412. [Lie05] E. Lieberman, C. Hauert, M. A. Now ak, Evolutionary Dy namics on Graphs , Nature 433 (2005), 312-316. [Lin94] K. Lindgren and M. N ordahl, Evolutionary dynamics of spatial games , Phys. D 75 (1994), 292. [Lot20] A. J. Lotk a, J. Am. Chem. S oc. 42 (1920), 1595-1599. [Ma y 73] J . Ma ynard Smith and G. Price, The logic of animal conflict , Natu re 246 (1973), 15. [Ma y 87] J . Ma ynard S mith and J. Hofbauer, The “battle of the sexes”: A genetic mo del with limit cycle b ehavior , Theoret. Po pulation Biol. 32 ( 1987), 1-14. [Mie07] J. Miekisz, Evo lutionary game th eory and p opulation d y namics (2007), arXiv:q- bio/07030 62 , Lectu re N otes in Math. (in press). [Mor62] P . A . P . Moran, The Statistical Processes of Evo lutionary Theory , (Clarendon, Oxford, 1962). [Mur04] J. D. Murray , Mathematical Biology I , Springer (2004). [Nas51] J. Nash, Non-co operative games , An n. Math. 54 (1951), 287-295. [Nei04] D. B. Neill, Evolutionary stability for large p opu lations , J. Theor. Biol. 227 (2004), 397-401. [Neu53] J. von Neumann and O. Morgenstern Theory of games and economic behavior , Princeton U niversi ty Press (1953). [No w92] M. A. Now ak and R. M. Ma y , Evolutionary games and spatial chaos , Nature 395 (1992), 826. [No w04] M. A. Now ak, A. Sasaki, C. T a ylor, and D. F uden berg, Emergence of co operation and evolutionary stability in fi nite p opulations , Nature 428 (2004), 646-650. [Oht06] H. O htsuki, C. Hauert, E. Lieberman, and M. Now ak , A simple rule for the evol ution of co op eration on graphs , Nature 441 (2006), 502-505. [Pis97] Y u. M. Pis’mak, Solution of the master equation for the Bak-S nepp en mo del of biological evolution in a fin ite ecosystem , Phys. Rev. E 56 (1997), 1326-1329. [Rei06] T. Reichen bac h, M. Mobilia, and E. F rey , Co existence versus extinction in the stochastic cyclic Lotka -V olterra mo del , Phys. Rev. E 74 (2006), 051907. [Sin96] B. Sinervo and C. M. Lively , The rock-pap er-scissors game and the evo lution of alternativ e male strategies , Nature 380 (1996), 240-243. COEVOLUT IONAR Y DYNAMICS IN FINITE AND INFINITE POPULA TIONS 15 [Sza98] G. Szab´ o and C. T˝ oke, Evolutionary p risoners d ilemma game on a square lattice , Phys. Rev. E 58 (1998), 69-73. [Sza99] G. Szab´ o, M. A. Santos, and J. F. F. Mendes, V ortex dynamics in a three-state mod el un der cyclic dominance , Phys. Rev. E 60 (1999), 3776-3780 . [Sza04] G. Szab´ o, A. Szolnoki, and R. Izs´ ak, Ro ck-scissors-paper game on reg- ular small-w orld netw orks , Journal of Physics A 37 (2004), 2599. [Sza05] G. Szab´ o, J. V uko v, and A. Szolnoki, Phase diagrams for an evolutionary prisoner’s dilemma game on tw o-dimensional lattices , Phys. Rev. E 72 (2005), 047107. [Sza07] G. Szab´ o and G. F ath, Evolutionary games on graphs , Phys. Rep. 446 (2007), 97-216 . [Szo04] A. S zolnoki and G. Szab´ o, V ertex dynamics d u ring domain growth in three-state mod els , Phys. Rev. E 70 (2004), 027101. [Szn00] K. Szna jd-W eron and J. Szna jd, Opinion ev olution in closed communit y , Internat. J Mo dern Phys. C 11 (2000), 1157-1165 . [T a y78] P . D. T a ylor and L. B. Jonk er, Evo lutionary stable strategies and game d ynamics , Math. Biosci. 40 (1978), 145-156. [T a y04] C. T aylor, D. F udenberg, A. S asaki, an d M. A. N o w ak, Evolutionary game dy namics in finite p opulations , Bull. Math. Biol. 66 (2004), 1621-1644. [T a y06] C. T a ylor and M. A. Now ak, Ev olutionary game dynamics with n on-uniform intera ction rates , Theoret. P opulation Biol. 69 (2006), 243-252. [T ra 04] A . T raulsen and J. C. Claussen, Similarity based co op eration and spatial segrega- tion, Phys. Rev. E 70 (2004), 046128. [T ra 05] A . T raulsen, J. C. Claussen, and C. Hauert, Co evol utionary D ynamics: F rom Finite to I nfinite Populations , Phys. R ev . Lett. 95 ( 2005), 238701. [T ra 06a] A. T raulsen, J. C. Claussen, and C. H auert, Co evolutionary d ynamics in large, but finite p opu lations, Phys. R ev. E 74 (2006), 011901 . [T ra 06c] A. T raulsen, M. A. N o w ak and J. Pac heco, Sto c hastic dynamics of inv asion and fixation , Phys. R ev. E 74 (2006), 011909. [T ra 06b] A. T raulsen, J. M. P acheco, an d L. A. Imhof, Stochasticit y and evolutionary stabilit y , Phys. Rev. E 74 (2006), 021905. [T ra 07] A . T raulse n, Martin A. N o w ak and J. Pa checo , St o chastic pay off ev aluation increases t he temp erature of selection, J. Theor. Biol. 244 (2007), 349-356. [V ol26 ] V. V olterra, Mem. Accad. Lincei. 2 (1926), 31, translation in: A n imal Ecology , R. N. Chapman (McGraw Hill, 1931), 409-448. [Wil04] G. Wild and P . D. T a ylor, Fitness and evolutionary stability in game theoretic mod els of finite p opu lations , Pro c R . So c. Lond. Ser. B Biol. Sci. 271 ( 2004), 2345-2349 . [W ri31 ] S. W righ t, Evolution in Mendelian Populations , Genetics 16 (1931), 97. [Zam00] K. R. Zamudio and B. Sinervo, Polygyn y , m ate- guarding, and p osthumous fertiliza- tion as alternativ e male mating strategies , Pro c. Natl. Acad. Sci. USA 97 (2000), 14427. [Zee80] E. C. Zeeman, Population dy namics from game theory , in: A. Nitecki, C. Robinson (Eds.), Pro ceedings of an International Conference on Global Theory of Dynamical Systems, Lecture Notes in Math. 819, Springer, Berlin, 1980. [Zee81] E. Zeeman, Dynamics of the evolution of an imal conflicts , J. Theor. Biol. 89 (1981), 249-270.