Distributed Agreement in Dynamic Peer-to-Peer Networks

Distributed Agreemen t in Dynamic P eer-to-P eer Net works ∗ John A ugustine † Gopal P andurangan ‡ P eter Robinson § Eli Upfal ¶ Abstract Motiv ated b y the need for r o bust and fas t distributed computation in highly dynamic Peer-to- Peer (P2P) netw orks, we study algorithms for the fundamen tal distributed agreement problem. P2P netw orks ar e highly dynamic net w orks that exp e rience heavy no de churn (i.e., no des join and lea v e t he net w ork con tinuously o ver t ime). Our goal is t o design fast a lgorithms (running in a small num ber of rounds) that guarantee, despite high no de c h urn rate, that almos t all no des reach a stable ag reement. Our main contributions are randomized distr ibuted algo rithms that guarantee stable almost-every wher e agr e ement with high pr obability even under high adv ersarial ch urn in a p olyloga r ithmic num b er of rounds. In particular , we present the following res ults: 1. An O (log 2 n )-round ( n is the stable netw ork size) rando mized algorithm that achiev es almost-everywhere agreement with high probability under up to line ar ch urn p er r ound (i.e., εn , for some small co nstant ε > 0), assuming that the ch ur n is controlled by an oblivious adversary (that has complete knowledge and control o f what no des join and leav e and a t what time a nd has unlimited computationa l power, but is oblivious to the random c hoices ma de b y the alg orithm). Our algo rithm requires only po lylogarithmic in n bits to b e pro cessed and sent (p er ro und) by eac h node. 2. An O (log m log 3 n )-round randomized algorithm that a chi eves almost-everywhere agree- men t with hig h probability under up to ε √ n ch urn p er round (for some small ε > 0), where m is the size of the input v alue do ma in, that works ev en under an adaptive adv er- sary (that a lso knows the past r a ndom choices made by the a lgorithm). This a lgorithm requires up to po lynomial in n bits (a nd up to O (log m ) bits) to b e pro cessed and sent (per round) b y each no de. Our algorithms are the ﬁrs t-known, fully-distributed, agreement a lgorithms that work under highly dyna mic settings (i.e., high chu rn rates per step). F urthermore, they are lo ca lized (i.e., do not requir e any globa l top olog ical knowledge), simple, and easy to implement. These a lgorithms can s e r ve as building blo cks for implementing other non-trivial distributed computing tasks in dynamic P2P netw or ks. ∗ A preliminary version of this p aper app eared in the Pro ceedings of the AC M/SIAM Symp osium on Discrete Algorithms (SODA), 2012, 551-569. † Department of Computer Science and En gineering, Indian Institute of T ec hn ology Madras, Chennai, India. E-mail: augustine@ cse.iitm. ac.in . W ork d one while at the Division of Mathematical Sciences, Nany ang T echno- logical Un iv ersit y , Singap ore 637 371. ‡ Division of Mathematical Sciences, Nany ang T ec h nologica l Universit y , S ingapore 637371 and D ep artmen t of Computer S cience, Brown Universit y , Box 1910, Providence, R I 02912, USA. E-mail: gopalpandurangan@g mail.com . W ork supp orted in part by the follo wing gran ts: N anyang T ec h n ologic al Universit y gran t M58110 000, Singap ore Ministry of Education (MOE) Aca demic Researc h F und (AcRF) Tier 2 gran t MOE2010 -T2-2-082, US N SF grant CCF-1023 166, and a gran t from the US-Israel Binational Science F oundation (BSF). § Department of Computer Science, National Universit y of Singapore. E- mail: robin son@comp. nus.edu.sg ¶ Department of Computer Science, Bro wn Universit y , Bo x 1910, Providence, RI 02912 , USA. E-mail: eli@cs.brown.edu 1 1 In tro duction P eer-to -p eer (P2P) computing is emerging as on e of the k ey net wo rking tec hnologies in recen t y ears with man y application systems, e.g., Skyp e, BitT orren t, Cloudmark etc. Ho wev er, many of these systems are not truly P2P , as they are not fully decen tralized — they t yp ically use hybrid P2P along with cen tralized int erv ention. F or example, Cloudmark [25] is a large spam detecti on system used by millions of p eople that op erates by main taining a hybrid P2P n etw ork; it u ses a cen tral authorit y to regulate and charge users for participation in the net w ork. A k ey r eason for the lac k of fu lly-distributed P2P systems is the diﬃcult y in designing highly robust algorithms for large- scale dynamic P2P net works. Indeed, P2P n et wo rks are highly dyn amic net w orks c haracteriz ed b y high degree of no de churn — i.e., no des con tinuously join and leav e the net work. C on n ectio ns (edges) may b e added or deleted at an y time and thus the top ology c hanges very dynamically . In fact, measuremen t stu d ies of real-w orld P2P net works [33, 40, 64, 65] show that the c hurn rate is quite high: nearly 50% of p eers in r eal-w orld net works can b e replaced within an hour. (Ho wev er, despite a large ch urn rate, these stud ies also s h o w that the total n um b er of p eers in the net w ork is relativ ely stable .) W e n ote that p eer-to-p eer algorithms hav e b een p rop osed for a wid e v ariet y of computationally c hallenging tasks su c h as collab orativ e ﬁltering [19], sp am detection [25], data mining [28], w orm detection and supp ression [55, 67], and priv acy pr otection of archiv ed data [38]. Ho wev er, all algorithms p rop osed for these problems ha ve no theoretica l guarantee s of b eing able to w ork in a net w ork with a d y n amically c h anging top ology and a linear ch urn rate p er round . This is a ma jor b ottlenec k in implemen tatio n and wide-spr ead use of these algorithms. In this pap er, w e tak e a step to w ards designing robust algorit hms for large-sca le dynamic p eer- to-p eer net w orks. In particular, w e study the fund amen tal distributed agreemen t problem in P2P net works (the formal pr ob lem statemen t and mo d el is giv en in Section 2). An eﬃcien t solution to the agreemen t problem can b e used as a buildin g blo c k for robust and eﬃcien t solutions to other problems as ment ioned ab o ve. Ho wev er, t he distributed agreemen t problem in P2P net w orks is c h allenging since the g oal is to guaran tee almost-everywher e agreemen t, i.e., almost all no d es 1 should reac h consensus, ev en un der high ch urn rate. The c h urn rate can b e as muc h as linear p er time step (r ound) , i.e., up to a constant fraction of the stable net wo rk size can b e r eplaced p er time step. Indeed, until recen tly , almost all the work kno wn in the literature (see e.g., [32, 44, 45, 46, 66]) ha ve add ressed the almost-ev erywhere agreemen t p roblem only in static (b oun ded-degree) net works and these app roac hes do not work for dynamic net works with c hanging top ology . Such app roac hes fail in dynamic net w orks where b oth no des and edges can c hange b y a large amount in eve ry round. F or example, t he work of Upfal [66] show ed ho w one can ac hieve almost-ev erywhere agreemen t under u p to a line ar n um b er — up to εn , for a suﬃcien tly s m all ε > 0 — of Byzant ine faults in a b ounded-degree expand er net work ( n is the net w ork size). The algorithm required O (log n ) rounds and p olynomial (in n ) num b er of messages; ho wev er, the lo cal computation required by eac h pro cessor is exp onen tial. F urthermore, the algorit hm requires kno wledge of the global top ology , since at the start, no des need to ha v e this in f ormation “hardco ded” . T he w ork of Kin g et al. [47] is imp ortant in the cont ext of P2P n et wo rks, as it was the ﬁrst to study scalable (p olylogarithmic comm u n icatio n and num b er of round s) algo rithms for distributed agreemen t (and leader election) that are toleran t to Byzan tine faults. How ev er, as p oin ted out by the authors, their algorithm works only for s tatic n et works; similar to Up f al’s algorit hm, the no d es require hardco ded information on 1 In sparse, b ounded-degree netw orks, an advers ary can alw a ys isolate some num ber of non-faulty no des, hence almost-ev erywhere is the b est one can hop e for in such netw orks [32]. 2 the net wo rk top ology to b egin with and th us the algorithm d o es not work wh en th e top ology c h anges. In fact, this work ([47]) raises the op en qu estion of whether one can design agreemen t proto cols that can w ork in h ighly dynamic net works with a large ch urn r ate. 1.1 Our Main R esults Our ﬁrst con tribu tion is a rigorous theoretical framew ork for the d esign and analysis of algorithms for highly dynamic distributed systems with c h urn. W e b r ieﬂy describ e the k ey ingredien ts of our mo del h ere. (Our m o del is d escrib ed in d etail in Section 2.) Essentia lly , w e mo d el a P2P net work as a b ounded-degree expander g raph w hose top ology — b oth no d es and edges — c an c h ange arbitrarily from round to roun d and is con trolled b y an adve rsary . Ho wev er, w e assume that the total num b er of no des in the netw ork is stable. The n um b er of no de c h anges p er r ound is called the churn r ate or churn limit . W e consider a c h u rn rate of up to some εn , where n is the stable n et work size. Note that our m o del is qu ite general in the sense that we only assume that the top ology is an expander at ev ery step; no other sp ecial prop erties are assumed. Indeed, expanders h a ve b een used extensiv ely to mo del dynamic P2P net w orks 2 in wh ic h the expander prop erty is preserv ed under insertions and deletions of no des (e.g., [52, 59]). Since w e do not make assumptions on how the top ology is preserve d, our mo del is applicable to all suc h expander-based net works. (W e note that v arious prior w ork on dynamic net work mod els mak e similar assumptions on pr eserv ation of top ological prop erties — such as connectivit y , expansion etc. — at ev ery step under d ynamic e dge insertions/deletions — cf. Section 1.3. The issue of ho w such prop erties are preserv ed are abstract ed a w ay from the mo del, whic h allo ws one to fo cus on the d ynamism. Indeed, this abstraction has b een a feature of most dynamic mo dels e.g., see the surv ey of [20].) W e stud y stable, almost-ev erywhere, agreemen t in our mo d el. By “almost-ev eryw here”, we mean that almost all no des, except p ossibly β c ( n ) n o des (where c ( n ) is the order of the c hurn and β > 0 is a suitably small constan t — cf. Section 2) should reac h agreemen t on a common v alue. (This agreed v alue must b e the inpu t v alue of some no de.) By “stable” we mean that the agreed v alue is preserv ed su bsequen tly after the agreemen t is reac hed. Our main con tr ib ution is the d esign and analysis of randomized distribu ted algorit hms that guaran tee stable almost-ev erywhere agree men t with h igh probabilit y (i.e., with pr obabilit y 1 − 1 /n γ , for an arbitrary ﬁxed constant γ > 1) eve n u nder high adv ers arial c hurn in a p olylogarithmic num b er of rounds. Our algo rithms also guaran tee stabilit y once agreemen t h as b een reac hed. In particular, w e present the follo wing results (the precise theorem statemen ts are giv en in th e resp ectiv e sectio ns b elo w ): 1. (cf. Section 4 ) An O (log 2 n )-round ( n is the s table netw ork size) randomized algorithm that ac hieve s almost -ev erywh ere agreemen t with high prob abilit y und er up to line ar c h urn p er r ound (i.e., εn , for some s m all constan t ε > 0), assuming th at the c hurn is con trolled b y an oblivious adve rsary (that h as complete kno wledge of what no d es join and lea v e and at what time, bu t is oblivious to the random c hoices made by the algo rithm). Our algorithm requir es only p olylogarithmic in n bits to b e pro cessed and sen t (p er round) b y eac h n o de. 2. (cf. Section 5) An O (log m log 3 n )-round randomized algorithm th at ac h ieves almost-ev erywhere agreemen t with high probabilit y under up to ε √ n c hurn p er r ound , for some small ε > 0, 2 Expander graphs ha ve b een used ex tensive ly as candidates to solv e th e agreemen t and related problems in b ounded d egree graphs even in static settings (e.g., see [32, 44, 45, 46, 66]). Here we show that similar expansion prop erties are beneﬁ cial in the more challengi ng setting of dyn amic netw orks. 3 that w orks ev en under an adaptive adv ersary (that also kn o ws the past random choic es made b y the algorithm). Here m r efers to the size of the domain of inp ut v alues. This algorithm requires u p to p olynomial in n bits (and u p to O (log m ) bits) to b e pro cessed and sen t (p er round) by eac h no de. 3. (cf. Section 6) W e also s h o w that no d eterministic algorithm can guaran tee almost-ev eryw here agreemen t (regardless of the num b er of rounds), ev en und er constan t c hurn rate. T o the b est of our kno w ledge, our algorithms are the ﬁ rst-kno wn , fully-distribu ted, agreemen t algorithms that w ork under highly dynamic settings. Ou r algorithms are lo calized (do not requ ir e an y global top ological kno wledge), simple, and easy to implemen t. Th ese algorithms can serv e as building blo c ks f or imp lemen ting other non-trivial distributed computing tasks in P2P net works. 1.2 T ec hnical Con tributions The main tec hn ical c hallenge that w e ha v e to ov ercome is designing and analyzi ng distributed algorithms in net works where b oth no des and edges can c han ge by a large amoun t. Indeed, w hen the ch urn rate is linear, i.e., sa y εn p er round , in constant (1 /ε ) n umb er of rounds the en tire net work can b e renew ed! W e d eriv e tec h niques for information spreading (cf. Section 3) for doing non-trivial distributed computation in suc h n et works. Th e ﬁ r st tec hn ique that we use is ﬂo o ding. W e s h o w that in an expander-based P2P n et wo rk even under linear c hurn rate, it is p ossible to sp read information by ﬂo o ding if suﬃcien tly many (a β -fraction of the order of the c h u rn) no des initiate the information spreading (cf. Lemma 3.1). In other w ords, ev en an adaptiv e adv ersary cannot “s uppress” more than a small fraction of the v alues. The precise statemen ts and pro ofs are in Section 3. T o analyze these ﬂo o ding tec hniqu es w e in trod uce the dyn amic distance, whic h d escrib es the eﬀectiv e distance b et w een t wo no des with resp ect to the causal inﬂu ence. W e deﬁn e the notions of inﬂuence sets and dynamic distance (or ﬂo o ding time) in dynamic n et works with no de c h urn. (Similar notions ha v e b een deﬁned for dynamic graphs with a ﬁxed set of no des, e.g., [48, 17]). In (connected) net w orks where the no des are ﬁxed, the eﬀectiv e diameter (e.g., [48]) is alw a ys ﬁnite. In the highly dyn amic setting considered here, h o wev er, the eﬀectiv e distance b etw een tw o no des migh t b e inﬁnite, th u s w e need a more r eﬁ n ed deﬁnition for inﬂuence set and dyn amic distance. The second tec hnique that we u s e is “supp ort estimation” (cf. Section 3.4). S upp ort estima tion is a randomized tec hnique th at allo ws us to estimate the aggregate coun t (or sum ) of v alues of all or a subset of no des in the net w ork. Supp ort estimati on is d one in conjun ction with ﬂo o ding and uses prop erties of the exp onential distrib u tion (similar to [26, 56]). Supp ort estimation allo ws us to estimate the aggregate v alue quite pr ecisely with high probabilit y eve n under linear c hurn. But this wo rks only for an oblivious adversary; to get similar r esults for the adaptiv e case, w e need to increase the amoun t of b its th at can b e pro cessed and sen t by a n o de in ev ery r ound. Apart from supp ort estimat ion, we also use our ﬂo o ding tec hniques in the agreemen t algorit hm for the oblivio us case (cf. Algorithm 2) to sw a y the decision one w ay or the other. F or the adaptiv e case (cf. Algorithm 3), we us e the v ariance pr op ert y of a certain pr obabilit y distribu tion to ac hiev e the same eﬀect with constan t probabilit y . 4 1.3 Other Related W ork 1.3.1 Distributed Agreemen t The distributed agreemen t (or consensus) problem is imp ortan t in a wide range of app lications, suc h as d atabase managemen t, fault-tol eran t analysis of aggregate data, and co ordinated con trol of m ultiple agen ts or p eers. There is a long line of researc h on v arious ve rsions of the problem with man y imp ortan t results (see e.g., [7, 53] and th e references therein). Th e relaxation of achie ving agreemen t “almost ev erywhere” wa s introd uced by [32] in the conte xt of fault-tolerance in n et works of b oun ded degree wh ere all but O ( t ) no des ac h iev e agreement despite t = O ( n log n ) faults. This result was impr o ved b y [66], which sho w ed ho w to guaran tee almost ev erywhere agreement in the presence of a linear f raction of fault y nod es. Both th e w ork o f [32, 66] crucially u se expander graphs to sho w th eir r esu lts. W e also r efer to the related results of Berman and Gara y on the b utterﬂy net work [18]. 1.3.2 Byzan tine Agreemen t W e note that Byzan tine adv ersaries are qu ite diﬀeren t from the adve rsaries considered in this pap er. A Byzan tine adv ersary can hav e no d es b ehavi ng arbitrarily , bu t no new no des are added (i.e., no c hurn), whereas in our case (an external) adv ersary con trols the ch urn and top ology of the net work but not the b ehavio r of the no des. Despite this diﬀerence it is wo rth while to ment ion that there has b een signiﬁcan t work in designing p eer-to-peer net works that are pr o v ably robust to a large n um b er of Byzan tine faults [35, 42, 57, 62]. These fo cus only on r obustly enabling storage and retriev al of d ata items. The pr oblem of ac hieving almost-ev erywhere agreemen t among no des in P2P net works (mo deled as an exp and er graph) is considered by King et al. in [47] in the conte xt of th e leader election p roblem; essenti ally , [47] is a sparse (expander) net work implemen tati on of the fu ll information p rotocol of [46]. More sp eciﬁcally , [47] assu m es that the adv ersary corru pts a constant fraction b < 1 / 3 of the pro cesses that are un der its cont rol thr oughout the run of the algorithm. The proto col of [47] guaran tees that w ith constant probabilit y an uncorrupted leader will b e elected and that a 1 − O ( 1 log n ) fraction of the uncorru pted pro cesses know this leader. Again, w e note that the failure assu mption of [47] is quite diﬀerent from the one we use: Even though w e do not assume corru p ted no des, the adversary is free to sub ject d iﬀeren t no des to c h urn in eve ry round. Also note that the algorithm of [47] do es not w ork for dynamic n et works. Other works on handling Byzan tine n o des in the con text of P 2P netw orks include [62, 12, 34, 36, 14, 21, 68]. In [8], we h a ve dev eloped an almost-ev eryw here agreemen t algorithm that tolerates up to ˜ O ( √ n ) c hurn and ˜ O ( √ n ) c hurn p er round, in a dynamic net work mo d el. 1.3.3 Dynamic Net w orks Dynamic net works ha v e b een studied extensiv ely o ver the past th r ee decades. Some of the early studies fo cused on d ynamics that arise out of faults, i.e., when edges or no des fail. A num b er of fault mo dels, v aryin g according to exten t and n ature (e.g., probabilistic vs. w orst-case) and th e resulting dynamic n et works hav e b een an alyzed (e.g., see [7 , 53]). There ha ve b een sev eral studies on mo dels that constrain the r ate at w hic h changes o ccur, or assume that the n et work even tually stabilizes (e.g., see [1, 31, 37]). Some of the early w ork on general dynamic n et wo rks include [2, 11] wh ic h in trod uce ge neral building blocks for comm unication protocols on dynamic net works. Another 5 notable w ork is the lo cal balancing approac h of [10] for solving routing and m ulticommodity ﬂ o w problems on dynamic net w orks. Most of these pap ers dev elop algorithms that will w ork un der the assumption that the net work will ev ent ually stabilize and stop changing. Mo deling general dynamic net w orks has gained renew ed atten tion with the recen t adve n t of heterogeneous net w orks comp osed out of ad ho c, and mobile d evices. T o addr ess h ighly unpre- dictable net w ork dyn amics, stronger adve rsarial mo dels h a ve b een stud ied by [9, 27, 58, 50] and others; see the recent sur v ey of [20 ] an d the references therein. The works of [50, 9, 27] study a mo del in whic h th e comm un icatio n graph can change completely from one r oun d to another, w ith the only constrain t b eing that th e n et work is c onne cte d at e ach r ound ([50] and [27] also consider a stronger m o d el w here the constraint is that the net work should b e an expander or should h av e some sp eciﬁc expansion in eac h round ). The mo d el h as also b een applied to agreemen t prob lems in d yn amic netw orks; v arious v ersions of co ordinated consensus (where all no des must agree) ha v e b een considered in [50]. The recen t w ork of [24], stu d ies the ﬂo o ding time of Markovian evo lving dynamic graphs, a sp ecial class of evolvi ng graphs. W e note that the mo del of [49] allo ws only edge c h anges from r ound to roun d while the n o des remain ﬁxed. In this w ork, w e in tro duce a dyn amic net work m o del where b oth n o des and edges can c h ange b y a large amount (up to a linear fraction of the net w ork size). Therefore, the framework w e in trod uce in S ectio n 2 is more general than the mo del of [49], as it is additionally applicable to dynamic settings with no d e ch urn . Th e same is true for the notions of d ynamic d istance and inﬂuence set that we introdu ce in Sect ion 3.1, since in our mo del the dynamic distance is not necessarily ﬁn ite. In fact, acco rding to [48], coping with c hurn is one of the imp ortan t op en problems in the con text of dynamic net works. Our pap er tak es a step in this direction. An imp ortan t aspect of our algo rithms is that they will work and terminate correct ly eve n when the net work k eeps cont in ually c hanging. W e n ote that there has b een considerable p rior wo rk in dynamic P2P netw orks (see [59] and the references therein) but th ese do not assume that the net work k eeps con tinually c hanging o ver time. Due to the mobility of n o des, mobile ad -h o c n et works can also b e considered as dynamic net- w orks. The fo cus of [58] are the minimal requiremen ts that are nece ssary to correctly p erform ﬂo o ding and routing in h ighly dynamic n etw orks where edges ca n c hange b ut the set of no des remains the same. In the context of agreemen t pr oblems, elect ing a leader among mobile n o des that ma y join or lea ve the net w ork at an y time is the fo cus of [23]. T o mak e leader election solv able in this mo d el, Chung et al. in tro d uce the notion of D -connectedness, wh ic h ensur es information propagation among all no des that remain long enough in the net w ork. Note that, in con trast to our mo del, this assumption pr ohibits the adv ersary from p ermanen tly isolating p arts of the net work. The recen t w ork of [41 ] presen ts information spreading algorithms on dyn amic n et works based on net work cod ing [3]. 1.3.4 F ault-T olerance In most w ork on fault-toleran t agreemen t pr oblems the adversary a priori commits to a ﬁxed set of fault y no des. In contrast , [30 ] considers an adv ersary that can corrupt the state of some (p ossibly c h anging) set of O ( √ n ) no des in ev ery roun d. The median rule of [30] pr ovides an elegan t w a y to ensure that m ost no des stabilize on a common output v alue w ithin O (log n ) rounds, assuming a complete comm u nication graph. The median ru le, ho w ev er, only guarante es that this agreemen t lasts for some p olynomial num b er of r oun ds, whereas we are able to r etain agreeme n t ad inﬁnitum. Expander graphs and sp ectral prop erties h av e already b een applied extensiv ely to impro ve th e 6 net work d esign and fault-tolerance in d istributed computing (cf. [66, 32, 16]). La w and Siu [52] pro vide a distribu ted algorithm for main taining an expander in the presence of ch urn with high probabilit y by usin g Hamiltonian cycles. In [61] it is shown ho w to maint ain the expansion p rop ert y of a net work in the self-healing mo del where the adversary can delete/insert a n ew no d e in eve ry step. In the same m o del, [60] pr esen t a proto col that m ain tains constan t nod e degrees and constan t expansion (b oth with probability 1) against an adaptiv e adv ersary , wh ile requirin g only logarithmic (in the n et wo rk size) messages, time, and top ology c hanges p er deletion/i nsertion. In [6], it is shown that a S KIP graph (cf. [5]) conta ins a constant degree expand er as a subgraph with high p robabilit y . Moreo v er, it requires only constant o v erh ead for a no de to identify its inciden t edges that are part of this expander. Later on, [43] pr esen ted a self-stabilizing alg orithm th at con verges from an y w eakly connected graph to a SKIP graph in time p olylogarithmic in the n et work size, which yields a proto col that constructs an expander with high probabilit y . In [13] the authors introd uce the h yp erring, wh ic h is a s earch d ata structure supp orting insertions and deletions, while b eing able to handle concurr en t requests w ith lo w congestio n and dilation, wh ile guarant eeing O (1 / log n ) expansion and O (log n ) no de degree. The k -Flipp er algorithm of [54] transforms any und irected graph in to an expander (with high probabilit y) by iterativ ely p erforming ﬂ ip s on the end-ve rtices of p aths of length k + 2. Based on this proto col, the authors describ e how to design a p rotocol that s u pp orts d eletio ns and insertions of no des. Note that, ho wev er, the expans ion in [54] is only guaran teed w ith high probabilit y h o wev er, assum ing th at the n o de degree is Ω (log n ). Information spreading in distributed netw orks is the fo cus of [22] wh ere it is sh o wn that th is problem requ ires O (log n ) rounds in graphs w ith a certain conductance in the p ush/pull mo del where a no de can comm un icate with a randomly c h osen neigh b or in ev ery round. Aspnes et al. [4] co nsider information spreading via expander graph s aga inst an adv ersary , whic h is related to the ﬂo o ding tec hniques we derive in Section 3. More sp eciﬁcally , in [4 ] there are t w o opp osing parties “the alert” and “the wo rm” (cont rolled b y the adv ersary) that b oth try to gain con trol of the net w ork. In ev ery round eac h alerted no de can alert a constant num b er of its neighbors, whereas eac h of the wo rm n o des can infect a constan t num b er of non-alerted n o des in the net work. I n [4], Aspnes et al. show that th ere is a simple strategy to prev en t all b ut a small fraction of no des from b ecoming infected and, in case that the n et work has p o or expansion, the w orm w ill infect almost all no des. The work of [16] sho ws that, giv en a n et work that is initially an expander and assuming some linear fracti on of f aults, the remaining netw ork will stil l con tain a large comp onent with goo d expansion. These r esults are not directly app licable to dynamic net w orks with large amoun t of c hurn lik e th e ones we are considering, as the top ology migh t b e c h anging and linear ch urn p er round essent ially corresp onds to O ( n log n ) total c h u rn after Θ(log n ) round s—the min im u m amount of time necessary to solv e any non-trivial task in our mo del. In the con text of main taining prop erties in P2P net w orks, Kuhn et al. consider in [51] that up to O (log n ) no des can crash or join p er constant num b er of time steps. Despite this amoun t of c hurn, it is shown in [51] h o w to main tain a lo w p eer d egree and b ounded net work diameter in P2P systems by using the hyp ercub e and pancak e top ologies. Sche ideler and Schmid show in [63] ho w to main tain a distributed heap that allo w s join and lea ve op erations and, in addition, is resisten t to Sybil attac ks. A robust d istributed implemen tation of a distrib u ted hash table (DHT) in a P 2P net work is giv en b y [15], wh ic h can withstand t wo imp ortan t kind of attac ks: adaptiv e j oin-lea v e attac ks and adaptive insert/lo okup attac ks b y up to εn adverserial p eers. Note that, ho w ev er, that collisio ns are lik ely to o ccur once the num b er of attac ks b ecomes Ω( √ n ). 7 2 Mo del and Problem Statemen t W e are intereste d in establishing stable agreement in a dynamic p eer-to-p eer netw ork in whic h the no des and the edges change ov er time. The computation is structured into syn c hr onous rounds, i.e., we assume that no des r un at the same pro cessing sp eed and an y message that is sent by some no de u to its (curr ent) neigh b ors in some round r > 1 will b e receiv ed b y the end of r . T o en s ure scalabilit y , w e restrict the n um b er of bits sent p er round b y eac h n o d e to b e p olylogarithmic in the size of the input v alue domain (cf. Section 2.1). F or dealing with the muc h m ore p o w erfu l adaptiv e adv ers ary , we relax this requiremen t in Sections 3.5 and 5. W e mo d el dynamism in the net work as a family of undirected graphs ( G r ) r > 0 . A t the b eginning of eac h round r we start with the net w ork top ology G r − 1 . Then, the adv ersary get s to c hange the net work fr om G r − 1 to G r (in acco rdance to rules outlined b elo w). As is t ypical, an edge ( u, v ) ∈ E r indicates that u and v can comm unicate in round r by passing m essages. F or the sak e of r eadabilit y , we use V [ r,r + t ] as a sh orthand for T r + t i = r V i . Eac h no de u has a unique iden tiﬁer and is churne d in at some r ound r i and churne d out at some r o > r i . More p recisely , for eac h n o de u , there is a maximal range [ r i , r o − 1] suc h that u ∈ V [ r i ,r o − 1] and for ev ery r / ∈ [ r i , r o − 1], u / ∈ V r . An y information ab out the net w ork at large is only learned through the messages that u receiv es. It has n o a priori kno wledge ab out who its n eigh b ors will b e in the future. Neither do es u know when (or whether) it will b e c h u rned out. Note that we do not assume that n o des ha ve access to p erfect clo c ks, but we s ho w (cf. Section 3.3) how the no des can sync hronize their clocks. W e mak e the follo win g assump tions ab out the kin d of c hanges that our dyn amic n etw ork can encoun ter: Stable Netw ork Size: F or all r , | V r | = n , where n is a suitably large p ositiv e in teger. This assumption simpliﬁes our analysis. Ou r alg orithms will w ork correctly as long as the num b er of no des is r easonably s table (say , b et ween n − κn and n + κn for some su itably small constan t κ ). Also, w e assume that n (or a constan t factor estimate of n ) is common kn o wledge among the no des in the net work 3 . Ch urn: F or eac h r > 1, | V r \ V r − 1 | = | V r − 1 \ V r | 6 L = εc ( n ), where L is the churn limit , whic h is some ﬁxed ε > 0 fraction of the or der of the churn c ( n ); the equalit y in the ab o v e equation ensur es that the net work size remains stable. Our w ork is aimed at high lev els of c h u rn up to a c hurn limit L that is linear in n , i.e., c ( n ) = n . Bounded Degree Expanders: The sequence of graphs ( G r ) r > 0 is an expander family with a v ertex expansion of at least α , wh ic h is a ﬁxed p ositiv e constan t. 4 In other wo rds, the adv ers ary must ensure th at for ev ery G r and ev ery S ⊂ V r suc h that | S | 6 n/ 2, the n um b er of no des in V r \ S with a neighbor in S is at least α | S | . Note that w e do n ot explicitly consider the costs (comm un icatio n and computation) of main taining an expander und er c hurn. Instead, w e assume th at the duration of eac h time step in our mo del are n ormalized to b e large enough to encompass an expander main tenance proto col suc h as [52, 60]. 3 This assumption is imp ortant; estimating n accurately in our mo del is an in teresting problem in itself. 4 Note that the v alue of α determines ε , i.e. the fraction of churn that we can tolera te. In particul ar, to tolerate linear amount of ch urn , we require constan t exp ansion. In principle, our results can p otential ly b e extended to graphs with weak er expansion guarantees as well; h o w ev er the amount of c hurn that can b e tolera ted will b e reduced. 8 A run of a distributed algorit hm consists of an inﬁ nite num b er of round s. W e assume that the follo wing ev ents o ccur (in order) in ev ery round r : 1. A set of at most L no d es are c h u rned in and another set of L no d es are c hurned ou t. The edges of G r − 1 ma y b e c hanged as well, but G r has to ha v e a ve rtex expansion of at least α . These c hanges are und er the con trol of the adv ersary . 2. Th e n o des broadcast messages to th eir (current) n eigh b ors. 3. No des receiv e m essages b roadcast b y their neigh b ors. 4. No des p erform computation that can c hange their state and determine wh ich messages to send in round r + 1. Bounds on Parameters Recall that the c hurn limit L = εc ( n ), where ε > 0 is a constant and c ( n ) is the c hurn order. When c ( n ) = n , ε is the fraction of the no des c hurned out/in and therefore we require ε to b e less than 1 and must adhere to Equation (1). Moreo v er, we require the b ound β < 1 12 regarding th e right hand side of (1). Ho w ev er, wh en c ( n ) ∈ o ( n ), ε can exceed 1. In the remainder of this pap er, we consider β to b e a small constan t indep endent of n , suc h that (1) ε (1 + α ) α < β . It will b ecome app aren t in Section 3 that (1) presen ts a suﬃcien t co ndition for prev ent ing the adv ers ary from con taining the information propagated b y a set of β c ( n ) no d es. and that the churn exp ansion r atio ε (1+ α ) α presen ts a suﬃcient condition for information prop- agatio n in our mo del (c f. Lemma 3. 1). Finally , w e assum e that n is suitably la rge (cf. Equ a- tions 7 and 8). 2.1 Stable Agreemen t W e no w deﬁn e the Almos t Ever y where S t ab le A greement problem (or just the St able A gre ement p roblem for brevit y). Each n o d e v ∈ V 0 has an asso ciated input v alue from some v alue d omain of size m ; subsequen t new no des come with v alue ⊥ . Let V be the set of all input v alues associated with nod es in V 0 at th e start of round 1. Ev ery n o de u is equipp ed with a sp ecial decision v ariable deci sion u (initializ ed to ⊥ ) that can b e written at most once. W e sa y that a no d e u de cides on v al when u assigns v al to its decision u . Not e that this decision is irrevocable, i.e., ev ery no de can decide at most once in a run of an algorithm. As long as decision u = ⊥ , we sa y that u is unde cide d . St abl e Ag reement requires that a large fraction of the no des come to a stable agreemen t on one of the v alues in V . More precisely , an algorithm solves St able Ag reement in R r ounds , if it exhibits the follo w in g c haracteristic s in ev ery run, for any ﬁxed β adhering to (1). V alidit y: If, in some round r , no de u ∈ V r decides on a v alue v al , then v al ∈ V . Almost E v erywhere Agreement: W e say that the network h as r e ache d str ong almost every- wher e agr e ement by r ound R , if at least n − β c ( n ) no des in V R ha ve d ecided on the same v alue v al ∗ ∈ V and ev ery other nod e remains u ndecided, i. e., its decision v alue is ⊥ . In particular, no no de ev er decides on a v alue v al ′ ∈ V in the same run, for v al ′ 6 = v al ∗ . 9 Stabilit y: Let R b e the earliest round where n o des ha ve reac hed almost ev eryw h ere agreemen t on v alue v al ∗ . W e sa y that an algorithm r e aches stability by r ound R if, at every round r > R , at least n − β c ( n ) n o des in V r ha ve decided on v al ∗ . W e also consider a we ak er v arian t of the ab o v e problem th at we call Almost Ever y where Binar y Consensus (or simply , Binar y Consensu s ) where the input v alues in V are restricted to { 0 , 1 } . W e consider t wo t yp es of adv ersaries for ou r randomized algorithms. An oblivious ad versary m ust commit in adv ance to the entire sequence of graphs ( G r ) r > 0 . In other w ord s, an oblivious adv ers ary must commit indep endently of the r and om c hoices made b y t he algorithm. W e also consider the m ore p o w erf ul adaptive adv ers ary that can observe the en tire state of the netw ork in ev ery round r (including all the rand om choic es made until r ou n d r − 1), and then c ho oses the no des to b e c hurned out/in and ho w to change the top ology of G r +1 . F or the sak e of r eadabilit y , we treat log n as an in teger and omit the n ecessary ceiling or ﬂo or op erations if their applicatio n is clear from the conte xt. 3 T ec hn iques for Information Spreading In this section, we ﬁrst deriv e an d analyze tec hniqu es to spread information in the net w ork despite c hurn. First, w e sh o w that the adv ersary is unable to preve n t a suﬃcientl y large set of n o d es (of size at least β c ( n )) to propagate their information to almost all other no des (cf. Lemma 3.1). Building on this result, w e analyze the capabilit y of individual n o des to s pread their information. W e show in Lemma 3.2 and Corollary 3.3 that at most β c ( n ) no des can b e h indered by the adve rsary . Finally , w e sh ow in Lemmas 3.5 and 3.6 that there is a large set of no des V ∗ suc h that all no des in V ∗ are able to propagate their information to a large c ommon set of no des. In Sections 3.4 and 3.5, we describ e how to use the previously deriv ed tec hn iques on information spreading to estimate the “ s u pp ort” (i.e. num b er) of n o des that b elong to a sp eciﬁc category (either red or blue). These proto cols will f orm a fundamental b u ilding block for our St ab le A greement algorithms. Due to the high amoun t of c hurn and the dynamically c hanging net wo rk, we use message ﬂo o ding to disseminate and gather information. W e now precisely deﬁne ﬂo o ding. Any no de can initiate a message for ﬂ o o ding. Messages that need to b e ﬂo o ded hav e an indicator bit b Flood set to 1. Eac h of these messages also con tains a terminating condition. The initiating no de send s copies of the message to itself and its n eigh b ors. When a no de receiv es a message with bFl ood set to 1, it con tinues to send copies of that message to itself and its n eigh b ors in subsequ ent roun d s un til the terminating condition is satisﬁed. 3.1 Dynamic Distance and Inﬂuence Set Informally , the dynamic distance from no de u to no de v is the n um b er of rounds required f or a message at u to r eac h v . W e n o w formally deﬁne the notion of dynamic distanc e of a no de v fr om u starting at round r , denoted by DD r ( u → v ). When the sub script r is omitted, we assume that r = 1. Supp ose no de u joins the net w ork at round r u , and, from round max( r u , r ) on w ard, u initiate s a message m for ﬂo o d ing whose terminating condition is: h has reached v i . If u is c hurned out b efore r , then DD r ( u → v ) is u n deﬁned. Supp ose the ﬁr st of those ﬂo o ded messages reac hes v in roun d r + ∆ r . Then, DD r ( u → v ) = ∆ r . Note that this deﬁn ition allo ws DD r ( u → v ) to b e 10 inﬁnite u nder t wo scenarios. Firstly , no d e v may b e ch urn ed out b efore any cop y of m r eac hes v . Secondly , at eac h roun d, v can b e shielded by c h urn no d es that absorb the ﬂo o ded messages and are then remo v ed f rom the net w ork b efore they can p ropagate these messages an y further. T he inﬂuence set of a no de u after R rounds starting at round r is giv en by: Infl r ( u, R ) = { v ∈ V r + R : DD r ( u → v ) 6 R } . Note that we require Infl r ( u, R ) ⊆ V r + R . Intuiti v ely , w e wan t the inﬂu ence set of u (in this dynamic setting) to captur e the n o des curr ently in the net w ork that w ere inﬂ uenced b y u . Note ho wev er that the in ﬂuence set of a no de u is meaningful ev en after u is c h urned out. Analoguously , w e deﬁne Infl r ( U, R ) = ∪ u ∈ U Infl r ( u, R ) , for an y set of no des U ⊆ V r . If w e consider only a single no de u , an (adaptiv e) adversary can easily p rev ent the inﬂuence set of this n o de f rom ev er reac hing an y signiﬁcan t size b y simp ly s h ielding u with c hurn no des that are replaced in ev ery round. 5 3.2 Prop erties of Inﬂuence Sets W e now fo cus our eﬀorts on charac terizing inﬂuen ce sets. This w ill help us in un d erstanding ho w w e can use ﬂo o ding to spr ead inform ation in the net work. F or the m ost p art of this section we assume that the net work is con trolled by an adaptiv e adversa ry (cf. S ectio n 2.1). The follo wing lemma shows that the num b er of no des that are suﬃcien t to inﬂ uence almost all the no des in the net work is giv en b y th e c hurn-expansion ratio (cf. E qu ation (1)): Lemma 3.1. Supp ose that the adversary is adaptive. Consider any set U ⊆ V r − 1 (for any r > 1 ) such that | U | > β c ( n ) . Then, after T = 2 & log n − log c ( n ) − log( β − ε (1+ α ) α ) − 1 log(1 + α ) ' numb er of r ounds, it holds that (2) | Infl r ( U, T ) | > n − β c ( n ) . When c onsidering line ar churn, i.e., c ( n ) = n , the b ound T b e c omes a c onstant indep endent of n . On the other hand, when c onsidering a churn or der of √ n , we get T ∈ O (log n ) . Pr o of. Our pr o of assumes that r = 1 for simplicit y as th e argumen ts extend quite easily to arb itrary v alues of r . W e pro ceed in t wo p arts: First w e sho w that the no des in U inﬂuence at least n/ 2 no des in some T 1 rounds. More pr ecisely , we sh o w that | Infl ( U, T 1 ) | > n/ 2. W e use v ertex expansion in a straigh tforw ard manner to establish this part. Then, in the second p art we sho w that n o des in Infl ( U, T 1 ) go on to inﬂuence more than n − β c ( n ) no des. W e cannot u s e the ve rtex expansion in a straightfo rw ard manner in the second part b ecause the cardinalit y of the set that is expandin g in inﬂuence is larger than n/ 2. Rather, w e use a sligh tly more sub tle argument in wh ic h we u s e 5 An oblivious adversa ry can ac hieve the same eﬀect with constan t probability for linea r ch urn. 11 v ertex expansion going b ac kward in time. The s econd part requires another T 1 rounds. Therefore, the t wo parts toge ther complete the pro of w hen w e set T = 2 T 1 . T o b egin th e ﬁr st part, consider U ⊆ V 0 at the start of round 1 with | U | > β c ( n ). In r ound 1, up to εc ( n ) no d es in U can b e c hurned out. Subsequently , the remaining no des in U inﬂuence some no des outside U as G 1 is an expand er with vertex expansion at least α . More p recisely , we can sa y that (3) | Infl ( U, 1) | > ( β c ( n ) − εc ( n ))(1 + α ) . A t the start of roun d 2, the graph changes dynamically to G 2 . In particular, up to εc ( n ) no d es migh t b e c hurned out and they ma y all b e in I nfl ( U, 1) in the worst case . Ho wev er, the inﬂuenced set w ill again expan d . Therefore, | Infl ( U, 2) | cannot b e less than ( | Infl ( U, 1) | − εc ( n ))(1 + α ) > β c ( n )(1 + α ) 2 − εc ( n )(1 + α ) 2 − εc ( n )(1 + α ). O f course, there will b e more c h u rn at the start of round 3 follo wed b y expansion leading to: | Infl ( U, 3) | >  β c ( n )(1 + α ) 2 − εc ( n )(1 + α ) 2 − εc ( n )(1 + α ) − εc ( n )  (1 + α ) = β c ( n )(1 + α ) 3 − εc ( n ) 3 X k =1 (1 + α ) k . This cycle of c h u rn follo w ed b y expansion con tin ues and w e get the f ollo wing b oun d at the end of some round i : | Infl ( U, i ) | > β c ( n )(1 + α ) i − εc ( n ) i X k =1 (1 + α ) k = β c ( n )(1 + α ) i + εc ( n ) 1 − (1 + α ) i +1 α − εc ( n ) Therefore, after (4) T 1 = & log n − log c ( n ) − log( β − ε (1+ α ) α ) − 1 log(1 + α ) ' rounds, w e get (5) | Infl ( U, T 1 ) | > n/ 2. No w we m ov e on to the second part of the p ro of. Let T = 2 T 1 . If | Infl ( U, T ) | > n − β c ( n ), w e are done. Th erefore, for the sak e of a contra diction, assume that | Infl ( U, T ) | 6 n − β c ( n ). Let S = V T \ Inf l ( U, T ), i.e., S is th e set of no des in V T that were n ot inﬂuenced b y U at (or b efore) round T . Moreo ver, | S | > β c ( n ) b ecause we ha ve assumed that | Infl ( U, T ) | 6 n − β c ( n ). W e will start at round T and work our wa y bac kw ard. F or q 6 T , let S q ⊆ V q , b e the set of all v ertices in V q that, starting from round q , inﬂuenced some v ertex in S at or b efore round T . More pr ecisely , S q = { s ∈ V q : Infl q ( s, T − q ) ∩ S 6 = ∅} . 12 Supp ose that | S T 1 | > n/ 2. T hen S T 1 ∩ Infl ( U, T 1 ) 6 = ∅ , since | Inf l ( U, T 1 ) | > n/ 2 by (5). Consider a no de s ∗ ∈ S T 1 ∩ Infl ( U, T 1 ). Note that s ∗ w as inﬂuenced by U and wen t on to in ﬂuence some n o de in S before (or at) round T . Ho w ev er, b y deﬁnition, no no de in S can b e inﬂu enced by any no d e in U at or b efore round T . W e ha v e th us reac hed a con tradiction. W e are left with showing that | S T 1 | > n/ 2. W e start with S and wo rk our w a y b ac kwa rds. W e kno w that | S | > β c ( n ) > β c ( n ) − εc ( n ). W e wan t to compu te the cardinalit y of S T − 1 . W e ﬁrs t fo cus on an in termediate s et S ′ , whic h we deﬁne as S ′ = S ∪ { s ′ : ∃ ( s, s ′ ) ∈ E T } . Since G T is an expand er, | S ′ | > | S | (1 + α ). F urthermore, it is also clear that eac h no de in S ′ could inﬂuence some n o de in S . Notice that S ′ \ S T − 1 is the set of no des in S ′ that we re c h urned in only at the start of round T . Therefore, | S T − 1 | > | S ′ | − εc ( n ) > | S | (1 + α ) − εc ( n ) > ( β c ( n ) − εc ( n ))(1 + α ) − εc ( n ) = β c ( n )(1 + α ) − εc ( n )(1 + α ) − εc ( n ) . Con tin uin g to w ork our wa y bac kw ards in time, we get | S T − 2 | > β c ( n )(1 + α ) 2 − εc ( n )(1 + α ) 2 − εc ( n )(1 + α ) − εc ( n ) , Or more generally , | S T − i | > β c ( n )(1 + α ) i − εc ( n ) X 0 6 j 6 i (1 + α ) j = β c ( n )(1 + α ) i + εc ( n ) 1 − (1 + α ) i +1 α = β c ( n )(1 + α ) i − εc ( n )(1 + α ) i +1 α + εc ( n ) α . W e no w w ant the v alue of i for whic h | S T − i | > n/ 2 + εc ( n ) α > n/ 2 . In other w ord s, w e w an t a v alue of i su c h that β c ( n )(1 + α ) i − εc ( n )(1 + α ) i +1 α + εc ( n ) α > n/ 2 + εc ( n ) α , whic h is obtained when i = T 1 . Therefore, it is easy to see that if w e set T = 2 T 1 , w e get | S T 1 | > n/ 2, thereby completing the pr o of. 13 A t ﬁrst glance, it migh t app ear to b e coun terintuitiv e that the order of the b ound T decrease s with increasing c hurn. When the adv ersary has the b eneﬁt of c hurn that is linear in n , our b ound on T is a constan t, bu t when the adv ersary is limited to a c h u rn order of √ n , we get T ∈ O (log n ). This, h o wev er, turns out to b e fairly natural when we note th at the size of the set U of n o des that w e start out w ith is in prop ortion to the c hurn limit. W e sa y that a no de u ∈ V r is suppr esse d for R r ounds or shielde d by churn if | Infl r ( u, R ) | < n − β c ( n ); otherwise w e say it is unsuppr esse d . The follo wing lemma shows that giv en a set with cardinalit y at least β c ( n ) some no de in that set will b e unsup pressed. Lemma 3.2. Consider the adaptive adversary. L et U b e any subset of V r − 1 , r > 1 , such that | U | > β c ( n ) . L e t T b e the b ound derive d in L emma 3.1. Ther e is at le ast one u ∗ ∈ U such that for some R ∈ O ( T log n ) , u ∗ is unsuppr esse d, i.e., | Infl r ( u ∗ , R ) | > n − β c ( n ) . In p articular, when the or der of the c hurn is n , T b e c omes a c onstant, and we have R = O (log n ) . Before w e pro ceed with our k ey arguments of the pro of, w e state a prop erty of b ip artite graphs that w e will use subsequentl y . Prop e rty 1. L et H = ( A, B , E ) b e a bip artite gr aph in which | A | > 1 and e v ery v ertex b ∈ B has at le ast one neighb or in A . Ther e is a subset A ∗ ⊂ A of c ar dinality at most ⌈| A | / 2 ⌉ such that |{ b : ∃ a ∗ ∈ A ∗ such that ( a ∗ , b ) ∈ E }| > ⌈| B | / 2 ⌉ . Pr o of. (o f Pr op ert y 1) Consider eac h no de in A to b e a u n ique color. Color eac h n o de in B usin g the color of a neigh b or in A c hosen arb itrarily . No w partition B into maximal subsets of n o des with lik e colors. Consider the parts of the partition sorted in decreasing order of their cardin alitie s. W e no w greedily choose the ﬁrst ⌈| A | / 2 ⌉ colors in the sorted order of parts of B . W e call the c hosen colors C . Observe that colors in C co ve r at least as man y no des in B as those not in C . Sup p ose the colors in C co ve r few er than ⌈| B | / 2 ⌉ no des in B . Then the remaining colors will co v er ⌈| B | / 2 ⌉ , but that is a con tradiction. Therefore, colors in C cov er at least ⌈| B | / 2 ⌉ no des in B . Th e no des in A that ha ve the colors in C are the no d es that co mprise A ∗ , thereb y completing our pro of. Pr o of. (o f Lemma 3.2) Again, our pro of assumes r = 1 b ecause it generalizes to arb itrary v alues of r quite easily . F rom Lemma 3.1, we know that th e inﬂuence of all no des in U tak en together will reac h n − β c ( n ) no des in T round s. This do es not suﬃ ce b ecause we are in terested in sho wing that there is at least one n o de in V 0 that (individually) inﬂuences n − β c ( n ) n o des in V R for some R = O ( T log n ). F rom Lemma 3.1, we kno w that U (collectiv ely) will inﬂu ence at least n − β c ( n ) no des in T rounds, i.e., | Infl ( U, T ) | > n − β c ( n ). F rom Prop ert y 1, w e kn o w that there is a set U 1 ⊂ U of cardinalit y at most ⌈| U | / 2 ⌉ such that | Infl ( U 1 , T ) | > n − β c ( n ) 2 . Recalling that β < 1 12 < 1 3 , we kno w that | Inf l ( U 1 , T ) | > β c ( n ). W e can again u se Lemma 3.1 to say that Infl ( U 1 , T ) inﬂu ences more than n − β c ( n ) no des in additional T roun ds and, by trans itivit y , U 1 14 inﬂuences more th an n − β c ( n ) no des after 2 T rounds. W e therefore h a ve | Infl ( U 1 , 2 T ) | > n − β c ( n ). Again, we can choose a set U 2 ⊂ U 1 (using Prop erty 1 ) that consists of ⌈| U 1 | / 2 ⌉ no d es in U 1 suc h that | Infl ( U 2 , 2 T ) | > β c ( n ). Subsequent ly applying Lemma 3.1 extends the inﬂuen ce set of U 2 to more than n − β c ( n ) after 3 T rounds . In ev ery iteration i of the ab ov e argum en t, the size of the s et U i decreases by a constant f raction unt il w e are left with a single no d e u ∗ ∈ U such that | Infl ( u ∗ , O (log n ) T ) | > n − β c ( n ). Can β c ( n ) (or more no des) b e supp r essed for an y signiﬁcant n um b er of (sa y , Ω( T log n )) rounds? This is in immediate cont radiction to Lemma 3.2 b ecause an y such su ppressed set of no des must con tain an unsup pressed no de. This leads us to the follo wing corollary . Corollary 3.3. The numb e r of no des that c an b e suppr esse d for Ω( T log n ) r ounds is less tha n β c ( n ) , e v en if the network is c ontr ol le d by an adaptive adversary. Corollary 3.4. Consider an oblivious adversary that must c ommit to the entir e se quenc e of gr aphs in advanc e. If we cho ose a no de u uniformly at r andom fr om V 0 , with pr ob ability at le ast 1 − β c ( n ) n , then u wil l b e unsuppr esse d, i.e., | Infl ( u, Ω( T log n )) | > n − β c ( n ) . Pr o of. Let S ⊂ V 0 b e the set of no des suppressed for Ω( T log n ) rounds. Und er an oblivious adv ers ary , the no de u chose n un formly at random from V 0 will not b e in S with probabilit y 1 − β c ( n ) n , and hence, will not b e suppressed w ith that same probabilit y . The follo w ing t wo lemmas sho w that there exists a set V ∗ of un suppressed no des, all of wh ic h can inﬂuence a large common set of no des, giv en enough time. Lemma 3.5. Consider a dynamic network under line ar churn that is c ontr ol le d by an adaptive adversary. In some r ∈ O (log n ) r ounds, ther e is a set of unsuppr esse d no des V ∗ ⊆ V 0 of c ar dinality mor e than (1 − β ) n such that      \ v ∈ V ∗ Infl ( v , r )      > (1 − β ) n. Pr o of. Let V ∗ ⊆ V 0 b e an y set of unsupp ressed no des, i.e., in some c 0 log n r ounds for some constan t c 0 , the inﬂuence set of eac h v ∈ V ∗ has cardinalit y more than (1 − β ) n . Note that, h ow ev er, we c annot guaran tee that, for an y t wo v ertices v 1 and v 2 in V ∗ , | Infl ( v 1 , c 0 log n ) ∩ Infl ( v 1 , c 0 log n ) | > (1 − β ) n. Assume for simp licit y that | V ∗ | is a p o w er of 2. Consider an y pair of vertice s { v 1 , v 2 } , b oth mem b ers of V ∗ . Recalling that β < 1 12 < 1 3 , w e can sa y that | Infl ( v 1 , c 0 log n ) ∩ Inf l ( v 2 , c 0 log n ) | > β n. Therefore, considering that the inte rsected set Infl ( v 1 , c 0 log n ) ∩ In fl ( v 2 , c 0 log n ) of no des has cardinalit y at least β n , we can apply Lemma 3.1 leading to | Infl ( v 1 , c 0 log n + T ) ∩ Inf l ( v 2 , c 0 log n + T ) | > (1 − β ) n . W e can p artition V ∗ in to a set S 1 of | V ∗ | 2 pairs such that for eac h pair, the inte rsection of inﬂuence sets has cardinalit y more than (1 − β ) n after c 0 log n + T rounds. Similarly , we can 15 construct a set S 2 of quadruples b y disjointl y pairing th e p airs in S 1 . Usin g a similar argumen t, w e can sa y that for an y Q ∈ S 2 ,       \ v ∈ Q Infl ( v , c 0 log n + 2 T )       > (1 − β ) n. Progressing analogously , the set S log | V ∗ | will equ al V ∗ and w e can conclude that       \ v ∈ S log | V ∗ | Infl ( v , c 0 log n + T log | V ∗ | )       > (1 − β ) n. Since | V ∗ | 6 n , it h olds that c 0 log n + T log | V ∗ | ∈ O (log n ), thus completing the p ro of. Lemma 3.6. Supp ose that u p to ε √ n no des c an b e subje c te d to churn in any r ound by an adap- tive adversary. In some r ∈ O (log 2 n ) r ounds, th er e is a set of unsuppr esse d no des V ∗ ⊆ V 0 of c ar dinality at le ast n − β √ n such that      \ v ∈ V ∗ Infl ( v , r )      > n − β √ n. Pr o of. Since we assume that c ( n ) = √ n , the b ound T of Lemma 3.1 is in O (log n ). Therefore, b y instantia ting Corollary 3.3, we kno w that eac h of the unsuppr essed no des in V ∗ (whic h is of cardinalit y at least n − β √ n ) will inﬂu ence more th an n − β √ n no d es in O (log 2 n ) time. W e can u se the same argument as in Lemma 3.5 to sho w that in O (log n ) rounds, all th e uns u ppressed no d es ha ve a common inﬂu ence set of size at least Θ( n ). Th at common inﬂuence set will gro w to at least n − β √ n no des within another O (log 2 n ) r ou n ds. Thus a total of O (log 2 n ) r ou n ds is suﬃcien t to fulﬁll the requiremen ts. 3.3 Main taining I nformation in the Netw ork In a d ynamic n et wo rk with c hurn limit εn , the en tire s et of no des in the net w ork can b e c hurned out and new no des c h urned in w ithin 1 /ε r ounds. Ho w do the new n o des ev en know what algorithm is run ning? Ho w do they kn o w h o w far the algorithm h as progressed? T o add ress these basic questions, the netw ork n eeds to main tain some global information that is n ot lost as the no des in the net wo rk are c hurned out. Th ere are t wo basic pieces of information that need to b e main tained so that a new no de can join in and participate in the executio n of the distribu ted algorithm: 1. the algorithm that is cur ren tly executing, and 2. the num b er of rou n ds that hav e elapsed in the execution of the algorithm. In other wo rds, a global clo c k has to b e main tained. W e assume that the no d es in V 0 are all synchronized in their understanding of what algorit hm to execute and the global clock. The no des in the net work contin uously ﬂo o d information on wh at algorithm is running so that wh en a new no d e arriv es, unless it is s h ielded by ch urn, it receiv es this inf orm ation and can start participating in th e algorithm. T o mainta in the clo c k v alue, n o des send their current clock v alue to their immediate neighbors. When a new no de receiv es the clo c k information fr om a n eigh b or, it sets its o w n clock accordingly . Since n o des are not malicious or fault y , Lemma 3.1 ensures that inf ormation is correctly main tained in more than n − β c ( n ) no des. 16 3.4 Suppor t Estimation Under an Oblivious Adv ersary Supp ose we hav e a dynamic n etw ork with R no d es colored r ed in V 0 . R is also called the supp ort of red no d es. W e wa n t the no des in the netw ork to estimate R under an oblivious adversary . W e assume that the adve rsary c ho oses R and w h ic h R no des in V 0 to color r ed, bu t it do es not kno w the random choice s made by the algorithm. F urthermore, we assume that c h urn can b e linear in n , i.e., c ( n ) = n . Our algorithm u ses rand om num b ers dra wn from the exp onentia l distribution, w hose probab ilit y densit y function, w e r ecall, is parameterized by λ and giv en b y f ( x ) = λ exp( − λx ) for all x > 0. F urthermore, w e n otice that the exp ected v alue of a random num b er dr awn from the exp onentia l distribution of parameter λ is 1 /λ . W e n o w present t wo pr op erties of exp onent ial random v ariables that are cru cial to our con text. Consider K > 1 indep enden t rand om v ariables Y 1 , Y 2 , . . . , Y K , eac h follo wing the exp onen tial distribution of rate λ . Prop e rty 2 (see [39] for example) . The minimum among al l Y i ’s, for 1 6 i 6 K , is an e xp onen- tial ly distribute d r andom variable with p ar ameter K λ . The idea b ehind ou r alg orithm exploits Prop ert y 2 in th e foll o wing manner. If eac h of the R red no des generate an exp onenti ally distribu ted random num b er with parameter 1, then the minim um ¯ s among those R random num b ers will also b e exp onent ially distribu ted, but with parameter R . Th us 1 / ¯ s serv es as an estimate of R . T o get a more accurate estimation of R , w e exploit the follo win g prop ert y that p ro vides u s with sharp concen trati on w hen the pro cess is rep eated a suﬃcien t n um b er of times. Prop e rty 3 (see [56] and pp. 30, 35 of [29]) . L et X K = 1 K P K i =1 Y i . Then, for any ς ∈ (0 , 1 / 2) , P r      X K − 1 λ     > ς λ  6 2 exp − ς 2 K 3 ! . W e no w p resen t our algorithm for estimating R in p s eudo co de f orm at (assuming R > n/ 2); see Algorithm 1. Theorem 3.7. Consider an oblivious adversa ry and let γ b e a an arbitr ary ﬁxe d c onstant > 1 . L et ¯ R = max ( R , n − R ) . By exe cu ting Algorith m 1 to estimate b oth R and n − R , we c an estimate ¯ R to within [(1 − δ ) ¯ R , (1 + δ ) ¯ R ] for any δ > 2 β with pr ob ability at le ast 1 − n − γ . Pr o of. Wi thout loss of generalit y , let R > n/ 2. Out of the R red no d es up to β n no d es (c h osen obliviously) can b e suppr essed, lea ving u s with (6) R ′ > R − β n > (1 − 2 β ) R unsup p ressed red no des (since R > n/ 2). In a sligh t abuse of n otati on, w e use R and R ′ to denote b oth the cardinalit y and th e set of r ed no des and unsupp ressed red n o des, resp ectiv ely . W e deﬁne U = \ v ∈R ′ Infl ( v , t ); note that t = O (log n ) and | U | > (1 − β ) n (cf. Lemma 3.5). Let u b e some no de in U . Let V u = { v : v ∈ R ∧ u ∈ Infl ( v, t ) } . 17 Algorithm 1 Algorithm to estimate th e supp ort R of red no des when R > n / 2. The follo wing pseudoco de is executed at every no de u . P ∈ Θ(log n ) controls the precisi on of our estimate. Its exact v alue is w orked out in the proof of Theorem 3.7. At round 1: 1: Draw P random num b ers s 1 , s 2 , . . . , s i , . . . , s P , each from th e exp onential rand om distribution with rate 1. / / Each s i is chosen with a precision that ensures that the smallest p ossible p ositiv e v alue is at most 1 n Θ(1) ; / / Note that Θ(log n ) b its suﬃce. 2: F or each s i , create a message m u ( i ) containing s i and a terminating condition: h as encountered a message m v ( i ) with a smaller random number . / / Notice that a n o de u will ﬂo od exactly one message at eac h index i — in particular the smallest random num b er encountered by nod e u with message index i 3: F or each i , initiate ﬂoo ding of message m u ( i ). F or the next t = Θ(log n ) rounds: 4: Contin ue ﬂo oding messages resp ecting their termination conditions. / / It is easy to see that the num b er of bits transmitted p er round through a link is at most O (log 2 n ). At the end of the Θ(log n ) rounds: 5: F or each i , the nod e u holds a message m v ( i ). Let ¯ s u ( i ) b e the random number contained in m v ( i ). 6: ¯ s u ← P i ¯ s u ( i ) P . 7: No de u outputs 1 / ¯ s u as its estimate of R . / / Now that the estimation is completed, all messages can b e terminated. F or all u ∈ U , R ′ ⊆ V u ⊆ R . Notice that ¯ s u ( i ) computed by u in line num b er 6 of Algorithm 1 is based on r andom num b ers generated by all no des in V u . Therefore, at roun d t , no d e u is estimating R using the exp onen tial r andom num b ers that w ere dra wn b y no des in V u . Since our adv ersary is oblivious, the c h oice of V u is indep enden t of the c hoice of th e random num b ers generated by eac h v ∈ V u . Therefore, ¯ s u ( i ) is an exp onen tially distributed random num b er with rate | V u | > R ′ (cf. Prop ert y 2). F or any δ > 2 β , let ς 6 min { δ − 2 β 1 − δ , δ 1+ δ } . When P = 3 γ ln n ς 2 ∈ Θ(log n ) parallel iteratio ns are p erformed, where γ > 1, the required accuracy is obtained with probabilit y 1 − 1 Ω( n γ ) (cf. Prop ert y 3). 3.5 Suppor t Estimation Under an Adaptiv e A dv ersary The algorithm for supp ort estimation un der an oblivious adv ers ary (cf. Section 3.4) do es n ot w ork under an adaptive adv ersary . T o estimate the supp ort of red no des in the net w ork, eac h r ed no de dra ws a random num b er f rom the exp onentia l distribution and ﬂo o ds it in an attempt to spread the smallest rand om num b er. When the adv ersary is adaptive , the smallest random num b ers can easily b e targeted and suppr essed. T o mitigate this diﬃcult y , we consider a diﬀeren t algorithm in whic h the n um b er of bits comm unicated is larger. In particular, the n u m b er of b its comm unicated p er round b y eac h n o de executing this algorithm is at most p olynomial in n . Let R be the su pp ort of the red no d es. Every no de ﬂ o o ds its unique ident iﬁer along with a bit that indicates whether it is a red no d e or not. A t most β √ n no des’ identiﬁers can b e suppressed b y the adv ersary for Ω(log 2 n ) rounds lea ving at least n − β √ n unsupp ressed identi ﬁers (cf. Corollary 3.3). Eac h no d e coun ts the n u m b er of unique red identiﬁers A and non-red identiﬁers B that ﬂo o d o ve r it and estimates R to b e A + n − A − B 2 . This supp ort estimation tec hn ique generalizes quite easily to arbitrary c hurn ord er. Th erefore, 18 w e state the follo wing th eorem m ore generally . Theorem 3.8. Consider the algorithm mentione d ab ove in which no des ﬂo o d their unique identiﬁers indic ating whether they ar e r e d no des or not and assume that the network is c ontr ol le d by an adaptive adversary. L et c ( n ) b e the or der of the churn; we assume for simplicity that c ( n ) is either n or √ n . Then the f ol lowing holds: 1. A t le ast n − β c ( n ) no des estimate R b etwe en R − β c ( n ) 2 and R + β c ( n ) 2 . F urthermor e, these no des ar e awar e that their estimate is within R − β c ( n ) 2 and R + β c ( n ) 2 . 2. The r emaining no des ar e awar e that their estimate of R might fal l outside [ R − β c ( n ) 2 , R + β c ( n ) 2 ] . When c ( n ) = n , it r e quir es only O (log n ) r ounds, but when c ( n ) = √ n , i t r e qu i r es O (log 2 n ) r ounds. Pr o of. Let u b e any one of the n − β c ( n ) no des that receiv e at least n − β c ( n ) uns u ppressed iden tiﬁers (cf. Lemma 3.5 and Lemma 3.6). Let A and B b e the num b er of unique identiﬁers from red no des and non-red no d es, resp ectiv ely , that ﬂo o d ov er u . Let C = n − A − B 6 β c ( n ). This means that u estimates R to b e A + C 2 . Note that A 6 R 6 A + C and sin ce C 6 β c ( n ), R is estimated b et w een R − β c ( n ) 2 and R + β c ( n ) 2 . F urtherm ore, since u receiv ed n − β c ( n ) iden tiﬁers, it can b e sure that its estimate is b et ween R − β c ( n ) 2 and R + β c ( n ) 2 . If a no d e d o es n ot receiv e at least n − β c ( n ) iden tiﬁers, then it is a ware that its estimate of R migh t not b e within [ R − β c ( n ) 2 , R + β c ( n ) 2 ]. F rom Lemma 3.5, when c ( n ) = n , the algorithm tak es O (log n ) round s to complete b ecause w e wan t to ensure that u nsuppr essed no des ha v e ﬂoo ded the n et work. When c ( n ) = √ n , a s a consequence of Lemma 3.6, the algo rithm r equires O (log 2 n ) rounds. 4 St able A g reement Under an Oblivious A dve rsary In th is section w e will ﬁrst pr esen t Algorithm 2 for the simp ler problem of reac hing Binar y Con- sensus , wh ere the input v alues are r estricted to { 0 , 1 } (cf. Sect ion 2.1). W e will then use this algorithm as a subroutine for solving S t ab le A greement in Section 4.2. Throughout this section w e assume s uitable c hoices of ε and α suc h that the upp er b ound (7) β < 1 12 can b e satisﬁed for β ; n ote that (7) must hold in addition to b ound (1). Moreo v er, we assume that a n o de can send and pr o cess up to O (log 2 m ) bits in every round , where m is the size of the inpu t v alue domain. 4.1 Binar y Consensus A n o de u that executes Algorithm 2 pro ceeds in a sequence of O (log n ) chec kp oin ts that are in terlea v ed b y O (log n ) roun ds. Eac h no d e u has a bit v ariable b u that stores its current output v alue. A t eac h c heckpoint t i , no de u initiates supp ort estimation of the num b er of no des curr ently ha vin g 1 as their output bit b y using the algorithm describ ed in Section 3.4. (A t c heckpoint t R − 1 , no des estimate b oth: the supp ort of 1 and 0.) The outcome of this sup p ort estimation will b e a v ailable in c hec kp oin t t i +1 where u has d erived the estima tion #(1) . If u b eliev es that the sup p ort of 1 is small ( 6 1 4 n ), it sets its o w n output b u to 0; if, on the other h an d , #(1) is large ( > 3 4 n ), 19 u sets its output b u to 1. Th is guaran tees stabilit y once agreement has b een reac hed by a large n um b er of no d es. When the supp ort of 1 is roughly the same as the supp ort of 0, w e n eed a wa y to sw a y the decision to one side or th e other. T his is done b y ﬂo o ding th e netw ork whereby the ﬂo o ding message of no de v is w eigh ted by some randomly c hosen v alue, say r v . T he adv ersary can only guess which no de has the highest w eight and therefore, with consta n t probabilit y , the ﬂo o ding message with this highest weigh t (i.e., smallest r andom num b er) will b e u sed to set the outpu t bit b y almost all no des in the net work. Algorithm 2 Binar y Consensu s under an oblivious adv ersary; co de executed b y no de u . Let decision u b e initialized to ⊥ . Let b u b e the current output bit of u . If u ∈ V 0 , then b u is initialized to t h e input v alue of u ; otherwise it is set to ⊥ . Let t 1 = 1 be the ﬁrst c heckpoint round. Subsequen t c hec kp oin t rounds are given by t i = t i − 1 + O (log n ), for i > 1. F or the terminating chec kp oin t t R , we choose an R ∈ O (log n ), i.e., t R ∈ O (log 2 n ). At every c heckpoint round t i excluding t R : 1: In itiate sup p ort estimation (to b e completed in chec kp oint round t i +1 ). 2: Generate a random number r u uniformly from { 1 , . . . , n k } for suitably large but constant k . (With high p roba- bilit y , we wan t exactly one node to hav e generated min u r u .) 3: In itiate ﬂ oo ding of { r u , b u } with terminating cond ition: h ( has encountered another message initia ted by v 6 = u with r v < r u ) ∨ (current r ound > t i +1 ) i . At every c heckpoint round t i except t 1 : 4: Use the supp ort estimation initiated at chec kp oint round t i − 1 . Let #(1) b e u ’s estimated support v alue for t h e num b er of nodes that had an output of 1. 5: if #(1) 6 1 4 n then 6: b u ← 0. 7: el se if #(1) > 3 4 n then 8: b u ← 1. 9: el se if u has received ﬂo o ded messages initiated in t i − 1 then 10: Let { r v , b v } b e the message with the smallest random number that ﬂ oo ded ove r u . 11: b u ← b v . At terminating chec kpoint round t R : 12: i f #(1) > n 2 then 13: decision u ← 1. 14: Flo od a 1-decisio n message ad inﬁnitum. 15: e lse if #(0) > n 2 then 16: decision u ← 0. 17: Flo od 0-decision message ad inﬁnitum. If u receives a b -decision me ssage: 18: d ecision u ← b Theorem 4.1. A ssume that the adversa ry is oblivious and that the churn limit p er r ound is εn . A lgorithm 2 r e aches stability in O (log 2 n ) r ounds and achieves Binar y Consensus with high pr ob- ability. Pr o of. Throughout this pro of we rep eatedly inv ok e th e prop erties of the supp ort estimation as stated in Theorem 3.7, whic h su cceeds w ith probabilit y 1 − 1 /n γ . Assu ming that γ > 2, suﬃces to guaran tee that all of the Θ(log n ) inv o cations of the s u pp ort estimation are accurate with high probabilit y . 20 W e ﬁrst argue that V alidit y holds: Supp ose that all no des start with input v alue 1. Th e only w a y a no de can set its output to 0 is b y passing Line 5. This can happ en for at most β n n o des. The only wa y that more no d es can set their outp ut to 0 is if they estimate the sup p ort of 1 to b e in ( 1 4 n, 3 4 n ). I f β is su itably small, Theorem 3.7 guaran tees that with high pr obabilit y this will not happ en at an y no de. The argumen t is analogous for the case where all no des start with 0. Next w e sho w Almost E verywhere Agreemen t: Let N i b e the n um b er of no des at c hec kp oin t round t i that output 1. Let Low i , High i , and Mid i , resp ectiv ely , b e the sets of n o des in V t i for whic h #(1) 6 1 4 n , #(1) > 3 4 n , and 1 4 n < #(1) < 3 4 n ; note that no des are p laced in Low i , High i , and Mid i based on their #(1) v alues, whic h are estimates of N i − 1 , not N i . Clearly , w e h a ve that Low i + Mid i + High i = n . F or some i > 1, le t u ∗ ∈ V t i − 1 b e the no de that generated the smallest random num b er in c h eckpoint roun d t i − 1 among all no des in V t i − 1 . With high probabilit y , u ∗ will b e unique. By Corollary 3.4, with probab ility 1 − β (a constant) , u ∗ is unsu ppressed, implying that b u ∗ will b e used b y all no des in Mid i . Consider the follo wing cases: Case A ( N i − 1 6 ( 1 4 − δ ) n ): F rom Theorem 3.7, we kno w that with high probabilit y | Lo w i | > (1 − β ) n implying | Mid i | + | High i | 6 β n . Therefore, N i will cont in u e to b e very small leading to small estimates #(1) in s ubsequent chec kp oin ts. After O (log n ) c hec kp oin ts, this causes at least (1 − β ) n no des to decide on 0, with high probabilit y . Moreo v er, it is e asy to s ee that the remaining β n no des will not b e able to pass L ine 12, since the adv ersary cannot artiﬁcially increase the estimated supp ort of nod es with 1. (Recall fr om S ection 3.4 that by sup p ressing the minim um random v ariables, the adv ersary can only mak e the estimate smaller.) (W e are presen ting separate Cases B, C, and D for clarit y . Equiv alent ly , w e could ha ve treated them together as one case with the condition that ( 1 4 − δ ) n < N i − 1 < ( 3 4 + δ ) n leading to the implication that with h igh p robabilit y either | Low i | + | Mid i | > (1 − β ) n or | High i | + | Mid i | > (1 − β ) n .) Case B ( 1 4 − δ ) n < N i − 1 < ( 1 4 + δ ) n ): With high pr obabilit y , | Lo w i | + | Mid i | > (1 − β ) n implying | High i | 6 β n . Note ﬁrs t that no des in Low i will set their outpu t bits to 0. Since N i − 1 < ( 1 4 + δ ) n , there are at least ( 3 4 − δ ) n no des in V t − 1 that output 0. Of these, at m ost β n could ha ve b een suppr essed. So, with p robabilit y at least 3 4 − δ − β , u ∗ is an unsupp ressed no de that outputs 0. When u ∗ outputs 0, no des in Mid i will set their output bits to 0. Thus, considering Low i and Mid i , w e h a ve at least (1 − β ) n no d es that set th eir output bits to 0 with constant p r obabilit y . F or a suitably small δ and β < 1 4 − δ , this will lead to Case A in the next iteratio n, whic h means that subsequent ly n o des agree on 0. Case C ( ( 1 4 + δ ) n 6 N i − 1 6 ( 3 4 − δ ) n ): With h igh probabilit y , | Mid i | > (1 − β ) n . With constan t probabilit y (1 − β ), u ∗ will b e an unsup pressed no de and no des in M id i will set their output bits to the same v alue b u ∗ . This will lead to Case A in the next iteratio n. Case D ( ( 3 4 − δ ) n < N i − 1 < ( 3 4 + δ ) n ): This is similar to Case B, i.e., with constan t p robabilit y , at least (1 − β ) n no d es will reac h agreement on 1. Case E ( N i − 1 > ( 3 4 + δ ) n ): This is similar to Case A. With high p robabilit y , at least (1 − β ) n no des will decide on 1. Note that, when a c h ec kp oin t falls either und er Case A or Case E, with high probabilit y , it will remain in that case. When a c h ec kp oin t falls u nder C ase B, Case C, or C ase D, with constan t 21 probabilit y , w e get either Case A or Case E in the f ollo wing c hec kp oint. Therefore, in O (log n ) rounds, at least (1 − β ) n no des w ill reac h agreemen t with high probabilit y and all other no d es will remain undecided. F or prop ert y Stabilit y , note that if a no de has deci ded on some v alue in c h ec kp oin t t R , it con tinues to ﬂo o d its decision message. W e s ho wed that, w ith high pr obabilit y , at least (1 − β ) n no des w ill decide on the same bit v alue. Therefore, it follo ws by Lemma 3.1 that agreemen t will b e main tained ad in ﬁnitum among at least (1 − β ) n no des. 4.2 A 3-phas e Algorithm for St able A greement W e will no w describ e ho w w e use Algorit hm 2 as a buildin g block for s olving St able Agreement : In order to use Algorithm 2 to solv e St a ble Ag reement , w e w ill need to make a couple of crucial adaptations. • S u pp ose ev ery v ertex in V 0 has s ome auxiliary in f ormation. W e can easily adapt Algorithm 2 so that when a n o de u d ecides on a b it v alue b , then, it also inherits the auxiliary information of some v ∈ V 0 whose initial bit v alue w as b . This is guarante ed b ecause our algorithm ensures V alidit y . The auxiliary information can b e piggybac k ed on the messages that v generates throughout the course of the algorit hm. • F or a typica l agreemen t algorithm, w e assume that all no d es simulta neously start run n ing the algorithm. W e wa n t to adapt our algo rithm so that only no des in V 0 that ha v e an initial output b it of 1 initiate the algorithm, w hile n o des that start with 0 are considered passiv e, i.e., these no des do not generate messages themselv es, but still forward ﬂo o ding messages and start generating m essages from the n ext c h ec kp oint onw ard as so on as they notice that an instance of the algorit hm is running. W e now ske tc h h ow the algorithm can b e adapted: In the ﬁrst chec kp oin t t 1 , eac h no d e v with a 1 initiates supp ort estimation and ﬂo o d ing of message h r v , b v = 1 i . If the num b er of no des with 1 is small at chec kp oin t t 1 , then, at c h ec kp oin t t 2 , no des that receiv e estimate v alues will conclude 0, whic h will get reinforced in subsequent c hec kp oin ts. How ev er, if the n um b er of no des w ith a 1 at c hec kp oin t t 1 is large (in particular, larger than β n ), then, by suitable ﬂo o ding, at least (1 − β ) n n o des will kno w that a su pp ort estimatio n is u nderwa y and will participate from c hec kp oin t t 2 on ward. Selection and Flo o ding Phase: In th e very ﬁrst round, eac h no de u ∈ V 0 generates a uniform random n u mb er r u from (0 , 1) and , if the random num b er is less th an 4 log n n , u b ecomes a c andidate and initiates a m essage m u for ﬂo o ding. Th e message m u con tains the rand om num b er r u and the general v alue v al u (from domain { 0 , . . . , m } ) assigned to u by the adve rsary . This ph ase ends after Θ(log n ) r ounds to ensure that no more th an β n no d es are sup pressed (the pr ecise b ound on the num b er of rounds is giv en by Corollary 3.3). T he ﬂo o ding of the generate d m essages, ho w eve r, go es on ad inﬁnitum. Candidate Elimination Phase: W e initiate Θ(log n ) parallel iteratio ns of Binar y Consens us , whereby eac h iteration is asso ciated with one of th e Θ(log n ) ﬂo o ding messages, generated by the candidates in the ﬁrst phase. More precisely , the i -th instance of Binar y Consens us f or the i -th candidate and its ﬂ o o ding message m u i is designed as follo ws: no des that h a ve receiv ed a ﬂo o ded 22 message m u i , set their input bit (of the i -th instance of Binar y Consens us ) to 1 and initiate Binar y Consensus . W e sa y that a ﬂo o ded message m u is a survive d c andidate message if the instance of Binar y Consensus asso ciated with it reac h ed a decision v alue of 1. Conﬁrmation Phase: Among the sur vived candidate messages, ev ery no de v c h o oses the mes- sage m u j among its receiv ed messages that has the smallest r andom n um b er r u j (and asso ciated general input v alue v al u j ), and initiates a supp ort estimation for the num b er of no des that hav e receiv ed m u j . If the supp ort estimation rev eals a supp ort of at least (1 − β − δ ) n for m j then v decides on v al u j . No d es keep ﬂ o o ding their decision ad inﬁnitum. Theorem 4.2. Consider the oblivious adversary and supp ose that εn no des c an b e subje ct to churn in every r ound. The 3-phase algorithm is c orr e ct with high pr ob ability and r e aches St able Agree- ment in O (log 2 n ) r ounds. Pr o of. V alidit y f ollo ws immediately fr om the fact that no des only decide on some v alue that wa s the input v alue of a (surviv ed) candidate. W e now argue Almost Ev erywhere Agreemen t: Since all n o d es c h o ose ind ep endent ly whether to b ecome candidate, a s imp le app licati on of a standard Ch ernoﬀ b ou n d shows that the num b er of candidates is in the r ange [2 log n, 8 log n ] with probabilit y > 1 − n − 3 ; in the remainder of this pro of, w e condition on this even t to b e true. Consider the message m u generated b y some candid ate u in the S electi on and Flo o ding ph ase, and consider its asso ciated in stance of Binar y Cons ensus : If m u has reac hed at least (1 − β ) n no des b y ﬂo o ding, it follo ws by the prop erties of the Binar y Consensus algo rithm that the decision v alue of Binar y Consensus will b e 1 with p robabilit y 1 − n − 2 . On the other hand, if m u has a ve ry small supp ort (say , β n ), the consensus v alue will b e 0 with probabilit y 1 − n − 3 (cf. Case A of the pr o of of Theorem 4.1), and, if the supp ort of m u is neither too small n or to o large, the no d es will reac h consensus on either 0 or 1. Thus we can inte rpret a decision of 1 regarding the i -th message, as a conﬁrmation th at the i -th candidate had suﬃcien tly large supp ort. By taking a un ion b ound, it follo w s that, with p robabilit y at least 1 − n − 2 , at least (1 − β ) n no des agree on the set of su r viv ed candidate m essages, since th ey reac hed agreemen t in eac h iteration of Binar y Consensus . S ince the adve rsary is oblivious, eac h of the Θ(log n ) ﬂo o din g messages generated by the candid ates will not b e suppr essed with probability at least (1 − β ) (cf. Corollary 3.4). Therefore, with p robabilit y > 1 − n − 2 , at least one candidate u w ill hav e | Infl ( u, O (log n )) | > (1 − β ) n and th us the set of su rviv ed candidates S will b e nonempty; let w ∈ S b e the candidate wh o generated the smallest random num b er. When the supp ort estimation is initiated in the third p hase, a set of at least (1 − β ) n no des will measure w ’s supp ort to b e at least (1 − β − δ ) n for some δ > 2 β with probabilit y > 1 − n − 2 (cf. Theorem 3.7) and decide on the v alue v al w of w , whereas no des that do not observ e high supp ort r emain undecided. This sho ws Almost E verywhere Agreement . Analogously to Algorithm 2, no des in S ﬂ o o d their decision messages, whic h are adopted b y newly incoming n o des. By virtue of L emma 3.1, the stabilit y pr op ert y is main tained ad inﬁn itum. The additional running time ov erhead of the ab ov e three phases excludin g Algorithm 2 is only in O (log n ). Th is complete s the pro of of the Th eorem. 23 5 St able A g reement Under an A daptiv e A dv ersary In this section we consider the St able Ag reement pr oblem while d ealing with a more p o w erf u l adaptiv e adversary . A t the b eginning of a r ound r , this adv ersary observes the en tire state of the net work and previous comm u nication b et w een n o des (including ev en p revious outco mes of rand om c h oices!), an d thus can adapt its choi ce of G r to m ake it m uc h more diﬃcult for n o des to ac hiev e agreemen t. It is in structiv e to consider the algorithms present ed in Section 4 in this conte xt. Both ap- proac h es are do omed to fail in the presence of an adaptiv e adv ersary: F or the S t abl e A greement algorithm, the exp ected n u m b er of no des that initiate ﬂo o ding in the ﬂo o ding ph ase is log n . Even though eac h of these no d es w ould ha v e expand ed its inﬂuence set to some constan t size by the end of the next round, the adaptiv e adv ersary can sp ot and immediately c h urn out all these no des b efore they can comm un icate with an yone else, th us n one of these v alues will gain any supp ort. Algorithm 2 fails for the simple reason that the adv ers ary can selectiv ely suppr ess the ﬂo o d ing of the smallest generated ran d om v alue z ∈ { 1 , . . . , n k } with attac h ed bit b z from ev er reac hing some 50% of the n o des, whic h instead might use a distinct minimum v alue z ′ (with an at tac hed bit v alue b z ′ 6 = b z ) to guide their output c h anges. T o coun ter the diﬃculties w e hav e ment ioned, we relax th e mo del. Firstly , we limit the order of the c h u rn to √ n . S econdly , we allo w m essages of u p to a p olynomial (in n ) num b er of b its to b e sent o ver a link in a single round. Under these relaxations, we can estimate the supp ort of red no des in the n et work simply b y ﬂo o ding all the un ique iden tiﬁers of the red and non-red n o des (cf. Theorem 3.8). Similarly to Section 4, w e will ﬁrst solv e Binar y Consensus under these assum ptions and then sho w h o w to implement St able Agreement . In this section w e assume that the n u m b er of no des in the net work is suﬃcien tly large, suc h that (8) n ≫ 4 β 2 . Moreo v er, we assume that ev ery no de can send and pro cess up to O ( n c + log m ) bits p er round , where c is a constan t and m is the size of the input domain. 5.1 Binar y Consensus W e no w describ e an algorithm for solving Binar y Consensus , whic h is similar in spirit to Algorithm 2. The main diﬀerence is the handling of the case where the su pp ort of the no des that output 1 is roughly equ al to the supp ort of the no des with output bit 0. In this case w e r ely on the v ariance of ran d om c h oices made by individual no des to swa y the balance of the sup p ort to wards one of the t wo s ides with constan t p robabilit y . First, we argue wh y this tec hnique do es not wo rk wh en the c hurn li mit is ω ( √ n ): In our algorithm w e hand le the case where the su pp ort of 0 and 1 is roughly equal, b y causing eac h no de to up date its curren t outpu t bit to the outcome of a (p r iv ate) unbiased coin ﬂ ip. The standard deviation that w e get for the sum of these individual random v ariables is O ( √ n ) and the ev ent where the balance is swa yed by O ( √ n ) o ccurs with constant p r obabilit y . But since the adv ersary is adaptiv e and has ω ( √ n ) c hurn to pla y with, it can immediately undo this fa v ourable im b alance b y ch urn ing out no des s u c h that the supp ort of 0 and 1 w ill y et again b e roughly equal. Theorem 5.1. A lgorithm 3 solves Binar y Consensu s with high pr ob ability and r e aches stability within O (log 3 n ) r ounds, in the pr esenc e of an adaptive adversary and up to ε √ n churn p er r ound. 24 Algorithm 3 Binar y Consensu s under an adaptiv e adv ersary; co de executed by no de u . Let decision u b e initialized to ⊥ . Let b u b e the current output bit of u . If u ∈ V 0 , then b u is initialized to t h e input v alue of u ; otherwise it is set to ⊥ . Let t 1 = 1 b e the ﬁrst chec kp oin t round . Subsequ ent c heckp oin t round s are given by t i = t i − 1 + O (log 2 n ), for i > 1, with t ime b etw een consecutive chec kp oin t rounds suﬃcient for unsuppressed no des to reach a common inﬂuence (cf. Lemma 3.6). F or the terminating chec kp oin t t R , we choose an R ∈ O (log n ), i.e., t R ∈ O (log 3 n ). At every c heckpoint round t i excluding t R : 1: In itiate sup p ort estimation (to b e completed in chec kp oint round t i +1 ). At every c heckpoint round t i excluding t 1 , t R : 2: Use the supp ort estimation initiated at chec kp oint round t i − 1 . Let #(1) b e the estimated supp ort v alue for nod es that output 1. 3: if supp ort estimation is not accurate within [ R − β √ n 2 , R + β √ n 2 ] then 4: Do nothing. 5: el se if #(1) < n 2 − β √ n 2 then 6: b u ← 0. 7: el se if #(1) > n 2 + β √ n 2 then 8: b u ← 1. 9: el se 10: if the outcome of an unbia sed coin ﬂ ip is heads then 11: b u ← 0. 12: els e 13: b u ← 1. At terminating chec kpoint round t R : 14: i f #(1) > n 2 then 15: decision u ← 1. 16: Flo od a 1-decisio n message ad inﬁnitum. 17: e lse if #(0) > n 2 then 18: decision u ← 0. 19: Flo od a 0-decisio n message ad inﬁnitum. If u receives a b -decision me ssage: 20: d ecision u ← b Pr o of. First consider the V alidit y prop erty: Supp ose that all no des start with input v alue 1. The- orem 3.8 guaran tees th at any no d e u that receiv es insu ﬃcien tly many identi ﬁers for supp ort esti- mation, will execute L in e 4 and therefore neve r set its output to 0. On the other hand, if u d o es receiv e suﬃcien tly man y samples, again Th eorem 3.8 ens u res th at it will alw a ys pass the if-chec k in Line 7. Th us, no no d e can eve r outpu t 0. The case where all no des start with 0 can b e argued analogously . Next, we w ill sho w that Algorithm 3 satisﬁes Almost Ev erywhere Agreemen t. Let N i b e the n um b er of verti ces at c hec kp oint round t i with output b it 1. Let Low i , High i , and Mid i , re- sp ectiv ely , b e the sets of no des in V t i for whic h # (1) 6 n/ 2 − β √ n 2 , #(1) > n/ 2 + β √ n 2 , and n/ 2 − β √ n 2 < #(1) < n / 2 + β √ n 2 ; note th at no des are p laced in Low i , High i , and Mid i based on their #(1) v alues, whic h are estimate s of N i − 1 , not N i . I n a slight abuse of notatio n, we u se Low i , Mid i , and High i to also refer to their resp ectiv e cardinalities. Clearly , w e ha ve that Low i + Mid i + High i = n. 25 F urthermore, observe that either Low i or H igh i will b e 0. Otherwise, w e will h a ve t wo n o des suc h that one estimate s N i − 1 b elo w n / 2 − β √ n 2 , while the other estimates it ab o ve n / 2 + β √ n 2 — a violatio n of Theorem 3.8. Consider the follo wing cases: Case A ( N i − 1 < n/ 2 − β √ n ): F rom Theorem 3.8, Low i > n − β √ n and all n o des in Low i will set th emselv es to output 0. On ce this case is r eac hed in some chec kp oin t, it will b e reac hed in all future c hec kp oin ts u n til t R with high p robabilit y . Th erefore, the algorithm guaran tee s Almost Everywhere Agreemen t on 0 in t R ; with high p robabilit y , n o des do not pass Line 14 in c heckpoint t R , th us n o no de will ev er d ecide on 1. Case B ( N i − 1 > n/ 2 + β √ n ): T his case is similar to Case A with the diﬀerence that almost all no des decide on 1. Case C ( n/ 2 − β √ n 6 N i − 1 6 n/ 2 ): Notice that High i = 0. Th erefore, (9) Low i + Mid i > n − β √ n. W e consider t wo sub cases: 1. In th is case, we assume that Low i is at least n/ 2 + β √ n . This will set N i < n/ 2 − β √ n putting the net work in Case A in the next c h ec kp oint. 2. In this case, w e assume that Low i < n/ 2 + β √ n . This implies that Mid i > n − Low i − β √ n > n/ 2 − 2 β √ n. The no des in M id i will c h o ose 1 or 0 w ith equal probabilit y . The num b er of no des that choose 0 is a binomial d istribution with mean Mid i 2 and standard deviation √ Mid i 2 . Clearly , with some constan t probabilit y , Mid i 2 + √ Mid i 2 or more no d es in the set Mid i will set themselv es to output 0. Therefore, with constan t probabilit y , N i < n − Low i − Mid i 2 − √ Mid i 2 < n − Low i − n − Low i − β √ n 2 − q n − Low i − β √ n 2 Clearly , N i < n 2 − β √ n if 3 β √ n < q n − Low i − β √ n, whic h means that Low i + β √ n < n − 9 β 2 n. 26 W e kno w that Lo w i < n 2 + β √ n . Therefore, N i < n 2 − β √ n if n 2 + 2 β √ n < n − 9 β 2 n, that is, 2 β < √ n  1 2 − 9 β 2  . In other wo rds, as long as (10) n > 4 β 2  1 2 − 9 β 2  2 , it holds with constan t probabilit y that N i < n 2 − β √ n, whic h will pu t the net w ork in Case A at the next c hec kp oin t r ound. Assu m ption (8) guaran- tees that Condition (10) is easily met. Case D ( n / 2 < N i − 1 6 n/ 2 + β √ n ): Usin g argumen ts similar to Case C, w e can show th at with constan t probabilit y , N i > n 2 + β √ n, thereb y , p utting the net work in Case B. Clearly , after O (log n ) c h ec kp oint rounds the net work will r eac h either Case A or Case B 6 with high probabilit y and hence ac hiev e Almost Ev erywh ere Agreement on either 0 or 1. F or prop ert y Stabilit y , note that if a no de has d ecided on some v alue 6 = ⊥ in chec k p oin t t R , it con tinues to ﬂ o o d its decision message. Since at least (1 − β ) n ha v e decided, it follo ws by Lemma 3.1 that an y no d es th at h a ve b een c hurned in will also decide on this v alue within a constan t n um b er of rounds, th us agreemen t will b e main tained ad inﬁnitum. 5.2 St able Agreement No w that w e hav e a solution for Binar y Consensus , we will sho w ho w to u se it to solv e St able A gre ement where no des ha v e input v alues from some set { 0 , . . . , m } , for m > 1. Giv en some input v alue v al we can w r ite it in the base-2 num b er system as ( b 0 , . . . , b log m ) where b i ∈ { 0 , 1 } , for 0 6 i 6 log m . W e call v al a gener al input value and b i a binary input v alue . The basic idea of the St able A greement alg orithm is to run an instance of the Binar y Con- sensus algorithm for eac h b i and then combine the agreed bits d 1 , . . . , d log m to obtain agreemen t on the general input v alues. W e n o w d escrib e our algorithm; the detailed p s eudo co de is p r esen ted in Algorithm 4. Consider the i -th iteratio n of Algorithm 4 and supp ose that d 1 , . . . , d i − 1 are the ﬁrst i − 1 decision v alues of the previous i − 1 iterations of the Binar y Consensus algorithm. W e say that a n o de u knows a gener al input value matching the ﬁrst i binary de cision values , if u 6 Due to Equation (8) w e know that Cases A and B exist. 27 has kno wledge of a some v al ∈ { 0 , . . . , m } that was th e in p ut v alue of some no d e v and th e ﬁrst i − 1 b its of v al are exactly d 1 , . . . , d i − 1 . W e denote the i -th bit v alue of a general v alue v al by v al [ i ]. Recal l that the Binar y Consensus algorithm executes the supp ort estimation routines dev eloped in Section 3.5 . W e s lightly mo dify th e supp ort estimation rou tine by requiring eac h no de u to also piggybac k its current general v alue v al u on to the m essage it generates for supp ort estimatio n. Moreo ver, when u ﬂo o d s the decision messag e of the Binar y Consen sus algo rithm, it also p iggybac ks v al u . Whenev er a no de v up d ates its curren t output bit v alue to b , this guaran tees that v has learned of a general v alue v al w that has b as its ﬁ r st b it. Th us v sets v al v to the new v alue v al w and c ho oses its n ext input v alue for the ( i + 1)-th iteration of the Binar y Consensus algorithm to b e the ( i + 1)-th bit of v al v . T his is formalized in the follo wing lemma: Lemma 5.2. Consider i ter ation i of the Binar y Consens us subr outine exe cute d in Algo rithm 4. If a no de u has a curr ent binary output value of b , then the i -th b it of v al u is b . Pr o of. W e w ill show the result by in duction ov er the iterations of the Binar y Consensus algo- rithm. In itially , in the ﬁ rst iteration, no d e u uses the ﬁ r st bit of its inp u t v alue v al u . No w supp ose that u sets its output bit to 1 − b at some p oin t durin g the ﬁrst iteration. W e say that u violates gener al valid ity . T here are tw o possible cases: In the ﬁr st case, u observed a suﬃcien tly large supp ort for 1 − b and thus receiv ed a supp ort estimation message generated by a no de v that had a curren t output bit 1 − b , w hile in the second case u receiv ed a decision m essage generated by v . In either case, it follo ws fr om the d escription of the algorithm that n o de v has piggybac k ed v al v on top of this message. I f v al v [ i ] = 1 − b , then v has u p dated its o wn output bit without up dating v al v , due to receiving some message from another no d e v ′ , and b oth no d es, v and v ′ , violate general v alidit y . By bac kw ards tra versing this chai n of no des that violate general v alidit y , w e ev en tually reac h a n o de w which h as set its outpu t bit v alue to 1 − b but v al w [ i ] = b , without ha vin g receiv ed a message from a no de that violates general v alidit y . A ccording to the Binar y Consensus algorit hm, w only sets its bit v alue to 1 − b if it has either observed suﬃ cient supp ort for 1 − b or receiv ed a decision message conta ining a v alue of 1 − b . In b oth cases, it follo ws f rom the d escription of the algorithm that w up dates v al w to the piggybac k ed general v alue, th e i -th bit of whic h is 1 − b , pr o viding a cont radiction. The ab o ve lemma guaran tees that we can combine the decision bits of the Binar y Consensus iteratio ns to get a general decision v alue that satisﬁes v alidit y . W e can therefore sh o w the follo wing theorem: Theorem 5.3. Supp ose that the network is c ontr ol le d b y an adaptive adversary who c an subje ct up to ε √ n no des to churn in every r ound. Ther e is an algorithm that solves St able A greement with high pr ob ability and r e aches stability in O (log m log 3 n ) . Pr o of. Almo st-ev erywhere agreemen t follo ws almost imm ediately from the fact that the Binar y Consensus algorithm satisﬁes almost-ev erywhere agreemen t; what remains to b e shown is that all except β √ n nod es decide: Note that it is possib le that a set S of β √ n nod es can remain undecided when running an ins tance of the Binar y Consensus algorithm. T he no d es in S will not prop ose an y v alues in the next iteration but will participat e in the supp ort estimat ion and the propagation of m essages. By the correctness of the Binar y Consen sus algorithm, all except β √ n no des ev en tually kno w the decision bit d i of th e i -th iteratio n. In th e n ext iteration, an y n o de v that kno ws the decision bit d i +1 , also knows a general v alue v al such that v al [ i + 1] = d i +1 and 28 Algorithm 4 Solving St able A greement using Binar y Consensus . Pseudo co de for no de u . 1: Su pp ose t hat no de u starts with an initial general input va lue v al u . 2: for i ← 0 to log m do 3: No de u initiates B inar y Consensus by prop osing the i -th bit of its cu rren t v al u . Recall that Binar y Consensus will be reac hed in O (log 3 n ) round s. 4: When participating in th e supp ort estimati on that is p art of Bina r y Consensus each no de u piggybacks its v al u on top of the support estimation message. 5: Let d i b e the decision returned by Binar y Consensus alg orithm. If no de u has decided on bit v alue d i ∈ { 0 , 1 } , then u has learned of a general input v alue v al where the i -th bit is d i : Nod e u u p dates its cu rrent v alue v al u by setting it to v al and ﬂo ods v al u along the decision message according to th e Bina r y Consensus algorithm. 6: If u did not d ecide in t h e Bina r y Consensus algorithm, then u do es not prop ose a v alue in th e ( i + 1)-th iteration. 7: If u did not decide in th e last iteration, it remains undecided. Otherwise, u returns the v a l u as its decision v alue and ﬂo o d s this v alue ad inﬁnitum. th us can p rop ose in the subsequent iteration. Th is holds regardless of whether v ∈ S and thus all except β √ n no des participate in eac h iteration. F or v alidit y , w e argue that Algorithm 4 maint ains the follo wing inv ariant at the end of ev ery iteratio n i : a no d e that is a w are of the decision (bit) v alues o f the ﬁr s t i r u ns of the Binar y Consensus sub r outine, has knowle dge of a general v alue matc hing the ﬁrst i binary decision v alues. By Lemma 5.2, it follo ws that if a no de u p r op oses a bit b in iteration i , then b is the i -th bit of some general input v alue v al . This gu arantee s that the sequence of decision bits corresp ond to some general input v alue and th us satisﬁes v alidity . Finally , w e observ e that the p ro of of stabilit y is id en tical to th e Binar y Conse nsus algorit hm, th us completing the p ro of. 6 Imp ossibilit y of a Deterministic Solution In this section w e show that there is no deterministic algorithm to solv e St able Agreement ev en when the c hurn is restricted to only a constan t num b er of no des p er round . As a consequence, randomization is a necessit y for solving S t ab le A greement . W e int ro duce some w ell kno wn standard notations (see [7, Chap. 5]) used for sho wing imp os- sibilit y results of agreemen t pr oblems. Th e c onﬁgur ation C r of the n etw ork at round r consists of • the graph of th e net work at that p oin t in time, and • the lo cal state of eac h no d e in the n etw ork. A sp eciﬁc run ρ of some St able Agreement algorithm A is en tirely determined by an in ﬁ nite sequence of conﬁgur ations C 0 , C 1 , . . . where C 0 con tains the initial state of th e graph b efore the ﬁrst round. Consider the inpu t v alue domain { 0 , 1 } . A conﬁguration C r is 1-valent (resp., 0-valent ) if all p ossible ru ns of A that sh are the common preﬁx u p to and including C r , lead to an agreemen t v alue of 1 (resp., 0). Note that this d ecision v alue refers to the decision of th e large ma jorit y of no des; strictly sp eaking, a small fr actio n of no des might remain und ecided on ⊥ . A conﬁgur ation is univalent if it is either 1-v alen t or 0-v alen t. Any conﬁguration th at is not univ alen t is called a bivalent conﬁguration. The follo wing observ ation follo w s immediately fr om the deﬁn ition of th e St able A greement problem. 29 Observ ation 1. Consider a bivalent c onﬁgur ation C r in r ound r r e ache d by an algorithm A that solves St able Agreement and ensur es Almost Eve rywher e A gr e ement. No no de in V r c an have de cide d on a value 6 = ⊥ by r ound r . Theorem 6.1. Supp ose that the se quenc e of gr aphs ( G r ) r > 0 is an exp ander f amily with maximum de gr e e ∆ . A ssume tha t the c hurn is limite d to at most ∆ +1 no des p er r ound. Ther e is no de- terministic algorithm that solves St able A gre ement if the network i s c ontr ol le d by an adaptive adversary. Pr o of. W e use an argument that is similar to the argument used in the pr o of that f + 1 rounds are requir ed for consensus in the presence of f faults (cf. [7, Chap. 5]). F or the purp ose of this imp ossibilit y pro of, w e r estrict th e inp ut domain of n o des to { 0 , 1 } and allo w arbitrary congestion on the comm u nication c hannnels. Moreo v er, w e assume that the top ology of the n et work is ﬁxed throughout the run . Thus the adve rsary can only “replace” no des at the same p osition by some other no des. F or the sake of con tradictio n, assume th at su ch a d eterministic algorit hm A exists that solv es St able A greement under the assumed settings. W e will p ro ve our theorem by inductiv ely constructing an inﬁnite run ρ of this algorithm consisting of a sequence of biv alen t conﬁ gur ations. By virtue of Observ ation 1 this allo ws us to conclude that no des d o not reac h almost ev erywhere agreemen t. T o establish the basis of our indu ction, we need to sho w that there is an initial b iv alent con- ﬁguration C 0 at the start of round 1. Assum e in contradic tion that there is no biv alen t starting conﬁguration. Let D 0 (resp. D 1 ) b e the conﬁguration where all no d es start w ith a v alue 0 (resp., 1); note that b y v alidit y the d ecision v alue must b e on 0 (resp., 1). C onsider the sequence of conﬁ g- urations starting at D 0 and ending at D 1 where the only diﬀerence b et w een any tw o conﬁgur ations adjacen t (in this sequence) is a sin gle bit, i.e., exactly 1 n o de has a diﬀeren t input v alue. Sin ce D 0 is 0-v alen t and D 1 is 1-v alen t, this imp lies that there are t w o p ossible starting conﬁgurations in this sequence, C 0 0 and C 0 1 , in which (i) th e inpu t v alues are the same for all but one no de u 0 , but (ii) C 0 0 is 0-v alen t wh ereas C 0 1 is 1-v alent . Consider the resp ectiv e one-round extension of C 0 0 and C 0 1 where the adv ersary simply c hurns out no de u 0 . Both successor conﬁgur ations C 1 0 and C 1 1 are indistinguishible for all other no des, in particular they ha ve n o wa y of kno wing what initial v alue w as assigned to u 0 , sin ce all witnesses h a ve b een remo ved b y the adv ersary . Th erefore, C 1 0 and C 1 1 m ust b oth b e either 0-v alen t or 1-v alen t, a con tradiction. This sho w s that there is an initial biv alen t conﬁ gur ation, thereb y establishing the basis for ou r in duction. F or the inductiv e step, we assume that the net w ork is in a biv alen t conﬁgu r ation C r − 1 at the end of round r − 1. W e will extend C r − 1 b y one round (guided by the adv ersary) that yields another biv alen t conﬁguration C r . Assu me for the sak e of a con tradiction th at ev ery p ossible one-round extension of C r − 1 yields a univ alen t conﬁgur ation. Without loss of generalit y , assum e that th e one-round extension γ where n o no de is ch urned out is 1-v alen t and yields conﬁguration C r 1 . S ince b y assumption C r − 1 w as biv alen t, there is another one-round extension γ ′ that yields a 0-v alen t conﬁguration C r 0 . Moreo v er, w e kno w that a nonempt y set S of size at most ∆+1 no des must ha ve b een sub ject to c hurn in γ ′ . (This is the only diﬀerence b etw een C r 0 and C r 1 — r ecall that the edges of the graph are stable throughout the r un.) Let S ′ b e a su b set of S and let γ S ′ b e the one-round extension of C r − 1 that we get when only no des in S ′ are c hurned out. Clearly , γ = γ ∅ and γ ′ = γ S . Cons id er th e lattice of all su c h one- round extension b ounded by γ and γ ′ that is giv en by the p o wer set of S . Starting at γ and moving 30 to wards γ ′ along some path, we m ust r eac h a one-round extension γ { v 1 ,...,v k } that yields a 1-v alen t conﬁguration D r 1 , wh ereas the next p oin t on this path is some one-round extension γ { v 1 ,...,v k +1 } that ends in a 0-v alen t conﬁguration D r 0 . T he only diﬀerence b et wee n these t wo extensions is that no de v k +1 is c hurned out in th e latter bu t not in the form er extension. No w consider the one-round extensions of D r 0 and D r 1 where v k +1 and all its neighbors are ch urn ed out, yielding D r +1 0 and D r +1 1 . F or all other no d es, D r 0 and D r 1 are indistinguishib le and therefore they m ust either b oth b e 0-v alent or b oth b e 1-v alent . This, h o wev er, is a cont radiction. Considering that expander graphs usually are assumed to hav e constan t d egree, Th eorem 6.1 implies that ev en if w e limit the c h u rn to a constant, th e adaptive adv ersary can still b eat any deterministic algorithm. 7 Conclusion W e ha v e in tro duced a no v el framew ork for analyzing h ighly dynamic distribu ted systems with c hurn. W e b eliev e that our mo del captures the core c haracteristics of such systems: a large amoun t of c h urn p er round and a constan tly changing net w ork top ology . F uture w ork inv olv es extending our mo d el to include Byzan tine n o des and corrup ted comm unication c hannels. F urthermore, our w ork raises some k ey questions: Ho w muc h c hurn can we tolerate in an adaptiv e sett ing? Ar e there algorithms that tolerate linear (in n ) c h u rn in an adaptiv e setting? W e show that w e can tolerate O ( √ n ) c h urn in an adaptive setting, but it tak es a p olynomial (in n ) num b er of comm un icatio n bits p er round. An in triguing problem is to reduce the n um b er of bits to p olyloga rithmic in n . While the main fo cus of this pap er wa s ac hieving agreemen t among no d es which is on e of th e most imp ortan t tasks in a distributed system, as a next step, it migh t b e w orth wh ile to in v estigate whether the tec h niques presen ted in th is pap er can serv e as u seful b uilding blo c ks for tac kling other imp ortan t tasks like aggreg ation or leader election in highly d ynamic n et works. References [1] Y eh uda Afek, Baruc h A w erbuch, and Eli Gafni. Applying s tatic net w ork p roto cols to dynamic net works. In F OCS’87 , pages 358– 370, 1987. [2] Y eh uda Afek, Eli Gafni, and Adi Rosen. The slide mec h anism with applications in dynamic net works. In A CM PODC , pages 35–4 6, 1992. [3] R. Ahlsw ede, N. Cai, S. Li, and R. Y eung. Net w ork information ﬂow. T r ansactions on Infor- mation The ory , 46(4) :1204– 1216 , 2000. [4] James Asp nes, Na vin R u stagi, and Jared Saia. W orm v ersu s alert: Who wins in a battle for con trol of a large-sca le n et work? In OPODIS , p ages 443–456, 2007. [5] James Asp nes and Gauri S hah. Skip graphs. In SODA , pages 384–3 93, 2003. [6] James Aspnes and Udi Wieder. T h e expansion and mixing time of skip graph s with applica- tions. In SP AA , pages 126–13 4, 2005. [7] Hagit A ttiy a and Jennifer W elc h. D i stribute d Computing: F undamentals, Simulations and A dvanc e d T opics (2nd e dition) . John Wiley Interscie nce, Marc h 2004. 31 [8] John A ugustine, Gopal P andurangan, and P eter Robinson. F ast b yzan tine agreemen t in dy- namic net works. In PODC , pages 74–83 , 2013 . [9] Chen A vin, Mic hal Ko uc k ´ y, and Zvi Lotk er. Ho w to explore a fast-c hanging wo rld (co v er time of a simple r andom w alk on evo lving graphs). I n ICALP , pages 121–132, 2008. [10] B. A w erbu c h and F. T . Leigh ton. I m pro v ed appro ximation algorithms for the multi-c ommo dit y ﬂo w problem and local comp etitiv e routing in dynamic n etw orks. In A CM STOC , p ages 487– 496, Ma y 1994. [11] Baruch A w erb uc h , Boaz Pa tt-Shamir, Da vid Pele g, and Mic hael E. Saks. Adapting to asyn- c h r onous dynamic net works. In STOC’92 , pages 557–570, 1992. [12] Baruch A w erbuc h and Christian Scheidele r. Group Spreading: A Proto col for Pro v ably Secure Distributed Name Service. In ICA LP , pages 183– 195, 2004. [13] Baruch A w erb u c h a nd Chr istian Sc heideler. Th e hyperr ing: a lo w -congesti on d eterministic data structure for distributed environmen ts. In SODA , pages 318–327, 2004. [14] Baruch A w erbu c h and Christian Sc heideler. Robu st random num b er generation for p eer-to- p eer systems. In OPODIS , pages 275–2 89, 2006. [15] Baruch A w erb uc h and Chr istian S c heideler. T ow ards a scalable and r obust DHT. The ory of Computing Systems , 45:2 34–26 0, 2009. [16] Amitabha Bagc h i, Anku r Bharga v a, Amitabh Chaudh ary , Da vid Epp s tein, and Ch ristian Sc heideler. The eﬀect of faults on net w ork expans ion. The ory Comput. Syst. , 39(6):9 03–9 28, 2006. [17] Herv´ e Baumann, Pierluigi Cr escenzi, and Pierre F raigniaud. Parsimonio us ﬂo o ding in dynamic graphs. In PODC , pages 260–2 69, 2009. [18] Piotr Berman and Juan A. Gara y . F ast consensu s in net works of b ounded degree. Distribute d Computing , 7(2):6 7–73, 1993. [19] John F. Canny . Collab orativ e ﬁltering with pr iv acy . I n IE EE Symp osium on Se c urity and Privacy , pages 45–57, 2002. [20] Arn aud Casteigts, P aola Flo cc hini, W alter Q u attrocio cc h i, and Nicola S an toro. T ime-v arying graphs and dyn amic net w orks. CoRR , abs/1012.0 009, 2010. Short ve rsion in ADHOC-NO W 2011. [21] M. C astro, P . Druschel, A. Ganesh, A. Rowstron, and D. W alla c h. Secure Routing for Struc- tured P eer-to- P eer O verla y Net w orks. In OSDI , pages 299– 314, 2002. [22] Keren C ensor Hillel and Hadas Shac hnai. P artial information spreading with application to distributed maxim u m co v erage. In Pr o c e e dings of the 29th A CM SIGA CT-SIGOPS Symp osium on Principles of Distribute d Comp uting , PODC ’10, pages 161–170, New Y ork, NY, USA, 2010. A C M. 32 [23] Hyun Chul Ch ung, Pe ter Robinson, and Jennifer L. W elc h. O p timal regional consecutiv e leader electio n in mobile ad-ho c net works. FOMC ’11, pages 52–61. ACM, 2011. [24] And rea C lemen ti, Riccardo Silv estri, and Lu ca T revisan. Information spr eading in dynamic graphs. In PODC , 2012. [25] W ebsite of Cloudmark In c. http:/ /cloudmark.com/. [26] Ed ith Cohen. S ize-est imation framework with applications to transitiv e closure and reac h abil- it y . J. Comput. Syst. Sci. , 55(3): 441–4 53, 1997. [27] A. Das Sarma, A. Mo lla, and G. Pandurangan. F ast distributed compu tation in d ynamic net works via random w alks. I n DISC , 2012. [28] S ouptik Datta, Kanish ka Bhaduri, Chris Giannella, Ran W olﬀ, and Hillol Kargupta. Dis- tributed data mining in p eer-to-p eer n et works. IEEE Internet Computing , 10(4):18–2 6, 2006. [29] A. Dembo and O. Z eitouni. Large deviations tec hniques and applications. Ele arn , 1998. [30] Benjamin Do err, Leslie Ann Goldb erg, Lorenz Minder, Thomas Sauerw ald, and Ch ristian Sc heideler. Stabilizing consensus w ith the p o w er of tw o c h oices. In SP A A , pages 149–158, 2011. [31] S hlomi Dolev. Self- stabilization . MIT Pr ess, Cambridge, MA, USA, 2000. [32] Cynthia Dw ork, Davi d Peleg , Nic holas Pipp enger, and Eli Upfal. F ault tolerance in netw orks of b ounded degree. SIAM J. Comput. , 17(5) :975– 988, 1988. [33] Jarr et F alkner, Mic hael Piatek, John P . John, Arvin d Krishnamurth y , and Thomas E. An- derson. Proﬁling a million user dht. In Internet Me asur ement Comfer enc e , pages 129–1 34, 2007. [34] Amos Fiat, Stev e Gribble, Anna Karlin, Jared Saia, and Stefan Saroiu. Dynamicall y F ault- T oleran t Cont en t Addressable Net w orks. In P r o c e e dings of the First Internationa l W orkshop on P e er-to-Pe er Systems , Cam bridge, MA, 2002. [35] Amos Fiat and Jared Saia. Censorship resistan t p eer-to-p eer co n ten t addr essable net w orks. In SODA , pages 94–10 3, 2002. [36] Amos Fiat, Jared Saia, and Maxw ell Y oung. Making c hord robust to b yzan tine attac ks. [37] E. Gafni and B. Bertsekas. Distributed algorithms for generat ing loop-free routes in n et wo rks with frequen tly changing top ology . IEEE T r ans. Comm. , 29(1 ):11â ĂŞ18, 1981. [38] Roxa na Geam b asu, T ada yoshi K ohn o, Amit A. Levy , and Henry M. Levy . V anish: Increasing data pr iv acy with s elf-destructing data. In USENIX Se curity Symp osium , pages 299–316 , 2009. [39] C.M. Grinstead and J.L. Snell. Intr o duction to pr ob ability . American Mathematica l S o ciet y , 1997. 33 [40] P . K rishna Gum madi, Stefan Saroiu, and Stev en D. Gribble. A measurement stud y of n ap- ster and gn utella as examples of p eer-to-p eer ﬁle sharing systems. Com puter Communic ation R eview , 32(1):8 2, 2002. [41] Bernh ard Haeupler and Da vid Karger. F aster information dissemination in dynamic net works via net work cod ing. In A CM P ODC , pages 381–3 90, 2011. [42] Kir s ten Hildru m and John K ubiato wicz. Asymptotically eﬃcien t approac hes to f ault-tolerance in p eer-to-peer net works. In DISC , volume 2848 of L e ctur e Notes in Computer Scienc e , p ages 321–3 36. Springer, 2003 . [43] Riko Jacob, Andr´ ea W. Richa, Christian Schei deler, Stefan Schmid, and Hanjo T¨ aubig. A distributed p olylogarithmic time algorithm for self-stabiliz ing skip graphs. In PODC , pages 131–1 40, 2009. [44] Bru ce M. Kapron, Da vid Kemp e, V alerie King, Jared Saia, and Vishal Sanw alani. F ast asyn - c h r onous b yzan tine agreemen t and leader election with full inform ation. A CM T r ansactions on Algorithm s , 6(4), 2010. [45] V alerie King and J ared Saia. Breaking the O ( n 2 ) bit barrier: Scalable byza n tine agreemen t with an adaptiv e adv ers ary . J. A CM , 58:18:1–18 :24, Ju ly 2011 . [46] V alerie King, Jared Saia, Vishal S an w alani, and Erik V ee. Scalable leader election. In SODA , pages 990–9 99, 2006. [47] V alerie King, Jared S aia, Vishal Sanw alani, and Erik V ee. T o w ard s secure and scalable com- putation in p eer-to-p eer net works. In F OCS , pages 87–98 , 2006. [48] F. Ku hn and R . Oshman. Dynamic netw orks: Models and algo r ithms. SIGA CT News , 42(1): 82–96 , 2011. [49] F abian K uhn, Nancy Lynch, and Rotem Os h man. Distribute d computation in dyn amic net- w orks. In A CM STOC , p ages 513–522, 2010. [50] F abian Kuhn , Rotem O shman, and Y oram Moses. Co ord in ated consensus in dynamic netw orks. PODC ’11, pages 1–10. A CM, 2011. [51] F abian Ku hn, S tefan Schmid, and Roger W attenhofer. T o w ards w orst-case c hurn resistan t p eer-to-p eer systems. Distribute d Computing , 22(4):2 49–26 7, 2010. [52] C. La w and K .-Y. Siu. Distributed construction of rand om expand er net works. In INFOCOM 2003 , v olume 3, p ages 2133 – 2143 vol.3 , marc h-3 apr il 2003. [53] Nancy L ync h . Distribute d A lgorithms . Morgan Kaufm an Pu blishers, Inc., S an F rancisco, USA, 1996. [54] Pete r Mahlmann and Christian Sc hin delhauer. P eer-to- p eer net works based on random trans- formations of connected r egular undirected graphs. In SP AA , pages 155–16 4, 2005. [55] Da vid J. Malan and Mic hael D. Smith. Host-based d etect ion of w orms through p eer-to-p eer co op eration. In Vija y A tluri and Angelos D. Kerom ytis, editors, WORM , pages 72–80. ACM Press, 2005. 34 [56] Damon Mosk-A o y ama and Dev avrat S hah. F ast d istributed algorithms f or computing separable functions. IEEE T r ansactions on Information The ory , 54(7):2 997–3 007, 2008. [57] Moni Naor and Udi Wieder. A simple fault tolerant distributed hash table. In IPTPS , pages 88–97 , 2003. [58] Regina O’Dell and Roger W attenhofer. I nformation dissemination in h ighly dyn amic graphs. In DIALM-P OMC , pages 104– 110, 2005. [59] Gopal Pandurangan, Prabhakar Raghav an, and Eli Upf al. Building lo w-diameter P2P net- w orks. In FOCS , pages 492–4 99, 2001. [60] Gopal Pa ndurangan, Pet er Robinson, and Amitabh T rehan. Dex: Self-healing expand ers . In IEEE IPD PS , 2014 . [61] Gopal P andurangan and Amitabh T r eh an . Xheal: localized self-healing usin g expand ers. In PODC , p ages 301– 310, 2011. [62] Ch ristian S c heideler. Ho w to sp read adve rsarial no des?: rotate! In STOC , pages 704–713, 2005. [63] Ch ristian Sc heideler and Stefan Schmid. A distributed and o blivious heap. In A utomata, L anguages and Pr o gr amming , v olume 5556 of L e ctur e Notes in Computer Scienc e , pages 571– 582. Springer Berlin / Heidelb erg, 2009. [64] S ubhabrata Sen and Jia W ang. Analyzing p eer-to-p eer traﬃc across large net wo rks. IMW ’02, pages 137–1 50, New Y ork, NY, US A, 2002. A CM. [65] Daniel S tu tzbac h and Reza Rejaie. Und ers tanding c hurn in p eer-to-p eer n etw orks. I MC ’06, pages 189–2 02, New Y ork, NY, US A, 2006. A CM. [66] Eli Up fal. T ole rating a linear num b er of faults in n et wo rks of b oun ded d egree. Inf. Comput. , 115(2 ):312– 320, 1994. [67] V asileios Vlac hos, Stephanos Andr outsellis-Theot okis, and Diomidis Spinellis. Securit y appli- cations of p eer-to-p eer net works. Comput. Netw. , 45:19 5–205 , Ju ne 2004. [68] Maxwe ll Y oung, Aniket Kate, Ian Goldb erg, and Martin K arsten. Practical Robu st Comm u- nication in DHT s T olerating a Byzant ine Adv ersary . In ICDCS , pages 263–27 2, 2010. 35

Distributed Agreement in Dynamic Peer-to-Peer Networks

Original Paper

Comments & Academic Discussion

Leave a Comment

Original Paper

Related Papers

Comments & Academic Discussion

Leave a Comment