Detection, Understanding, and Prevention of Traceroute Measurement Artifacts

Detecti on, Understand ing, and Prev en tion of T raceroute Measuremen t Artifacts F abien Viger a Brice Augustin a Xa vi er Cuv el l ier a Cl´ emence Magnien a Matthi eu Lat ap y a , ∗ Timur F r iedman a Renata T eixeira a a Universit´ e Pierr e et Marie Curie – CNRS, L ab or atoir e LIP6 Abstract T raceroute is widely used, fr om the diagnosis of net w ork problems to the assem blage of in ternet maps. Unfortun ately , there are a num b er of problems with traceroute metho dology , whic h lead to the inference of erroneous routes. This pap er studies particular structures arisin g in nearly all traceroute measuremen ts. W e c haracterize them as “lo ops”, “cycles”, and “diamonds ”. W e identi fy load balancing as a p ossible cause for the app earance of false lo ops, cycl es, and diamonds, i.e., artifac ts that do not represent the in ternet top ology . W e pro vide a new pu blicly-a v ail able traceroute, called Paris tr ac er oute , whic h, by con trolling the pac k et header con ten ts, pro vides a truer picture of the actual routes that pac k ets follo w. W e p erformed measur emen ts, from the p ersp ectiv e of a single source tracing tow ards multiple destinations, and P aris traceroute allo w ed us to show that man y of the particular structur es w e ob- serv e are in deed tr aceroute measuremen t artifacts. Key wor ds: traceroute, net w ork topology measurement , measuremen t artifact , load bala ncing 1 In tro duction Jacobson’s tr ac er oute [1] is one of the most widely used netw ork measuremen t to ols. It rep orts a n IP address for eac h net w ork- lay er device along the path from a source to a destination host in an IP netw ork. Net w ork op erators and ∗ Corresp ond ing author. Address: L IP6 – 104 a v en u e du Pr´ esident Kenn edy – 75016 P aris – F rance T el: + 33 (0)1 44 27 87 84, F ax: +33 (0)1 44 27 74 9 5 Email addr ess: Mat thieu.Lat apy@lip6 .fr (Matthieu Latap y). Preprint submitted to Elsevier researc hers rely on traceroute to diagnose netw ork problems and to infer prop- erties of IP net w orks, suc h as the top ology of the in ternet. This has led to an impressiv e amount of w ork in recen t y ears [2,3,4,5,6,7,8], in whic h traceroute measuremen ts play a cen tral role. Some author s hav e noticed that traceroute suﬀers from deﬁciencies that lead to the inference of inaccurate routes, in particular in the presence of load balancing routers [3,4,9]. Ho w ev er, no systematic study of these deﬁcienc ies has b een undertak en. Therefore, p eople dealing with traceroute measuremen ts curren tly hav e no c hoice but to interpret surprising features in traceroute mea- suremen ts as either characteristic s of the r outing or of the net w ork’s to p ology . This is supp orted by the common assumptions t ha t these deﬁcien cies ha v e a v ery limited impact, and that, in an y case, nothing can b e done to a v oid them. The core contribution of this pap er, whic h is a longer ve rsion of our earlier w ork [10], is to sho w that b oth of these assumptions a re fa lse. W e sho w that the wide presence of lo ad-balancing routers in the in ternet induces a v ariet y of artifacts in traceroute measureme nts, and w e pro vide a rigorous approa c h to b oth quantify and a void man y of them. More precisely , w e f o cus on three particular structures often encoun tered in traceroute measureme nts, whic h we categorize as “lo o ps”, “cycles”, and “di- amonds”. Using measuremen ts from a single source tracing to wards multiple destinations, w e sho w that man y instances of these structures are actually measuremen t artifacts resulting from load-ba la ncing routers. W e provide a new traceroute, called Paris tr ac er oute , 1 whic h con tro ls pa ck et header con- ten ts to largely limit the eﬀects of load balancing, and th us obtain a more precise picture o f the actual routes. W e sho w that man y of the o bserv ed struc- tures disapp ear when one uses Paris traceroute. Finally , w e explain most other instances using a dditio nal information provided by Paris traceroute, and sug- gest p ossible causes for the remaining ones. Throughout this pap er, we use data obtained by tr a cing routes from one pa r - ticular source to illustrate our results (see Sec. 3). F rom the outset, w e insist on the fact that this data is not mean t to b e statistically represen tativ e of what can b e observ ed on the interne t in general: it serv es as an illustration only , and the quan tities repo rted ma y diﬀer signiﬁcan tly from what w ould b e observ ed from other sources. Obtaining a represen tativ e view of the a v erage b eha vior of the t r a ceroute to ol w ould clearly b e of inte rest, but is out of the scop e of this pap er: we fo cus here on the iden tiﬁcation of traceroute artifacts, their rigorous interpretation, and their suppression using P aris traceroute. This pap er is structured as follow s. Sec. 2 describ es the classic traceroute 1 P aris traceroute is free, op en-source soft w are, a v ailable from http://w ww.paris - t raceroute.net/ . 2 to ol, its deﬁciencies, and the new to ol w e built to circum v en t these deﬁcien- cies, Paris traceroute. Sec. 3 desc rib es our methodo logical f r amew ork. Sec. 4 categorizes the particular structures that w e encoun ter and study in tracer- oute measure men ts. Sec. 5 and Sec. 6 study these structures, and provide o ur explanations f o r them. Sec. 7 discuss es r elated w ork. Finally , Sec. 8 presen ts our conclusions and p ersp ectiv es for future work. 2 Building a b etter traceroute This section describes the to ols used in this pap er to study traceroute mea- suremen t a rtifacts. Sec. 2.1 describ es the classic t r a ceroute to ol, and may b e skipped b y those familiar with it. Sec. 2.2 describ es the deﬁciencies in clas- sic traceroute in the f ace of loa d balancing. Sec. 2.3 then presen ts our new traceroute, P aris traceroute, which av oids some of these deﬁciencies, nota bly the ones induced by p er-ﬂow load balancing. 2.1 T r ac er oute There are man y v arieties and deriv a tiv es o f the traceroute to ol. This section describes a g eneric probing sc heme, based on Jacobson’s v ersion [1]. Related to ols, suc h as NANOG tr ac er oute [11] and sk itter [3], do v ery similar things. F or a go o d detailed description of ho w traceroute works , see Stev ens [12]. A t a high lev el, three para meters deﬁne an in v o cation of the traceroute to ol: the destination IP address, d , the pro b e pac k et proto col, P , a nd the n um b er of prob es p er hop, n . The address d may b e an y legal IP address. The proto col P is one o f either: UDP (by default), ICMP ( a lso fairly common), o r TCP (not used b y t he classic traceroute, but more and more used by o ther to ols, suc h as tcptr ac er oute [13], b ecause there is often less ﬁltering of TCP pac ke ts). Probing with traceroute is done hop b y hop, moving aw a y from the source to w ards the destination in a series of rounds. Each round is asso ciated with a hop c ount , h . The hop coun t starts at one a nd is incremen ted after eac h round un til the destination is reac hed, or un til ano ther stopping condition applies. A r ound of probing consists in se nding n prob e pack ets with proto col P tow ards destination d . By default n is equal to three . The prob e pac k ets are sen t with the v alue h in the IP time-to -liv e (TTL) ﬁeld. The TTL of an IP pac k et is supp osed to b e examined b y each router that the pac k et reac hes. If the TTL is greater than one, then it is decremen ted and the pac k et is forwarded. If it is equal to one, the router drops the pack et and 3 sends an ICMP Ti m e Exc e e de d message back t o the source [14]. Rout ers are required to emplo y , as the source address of this ICMP message, the address of the IP in terface that sends the ICMP pac k et [15, Sec. 4.3.2 .4]. 2 When routing is symmetric, this is t ypically the address of the inte rfa ce t ha t receiv ed the prob e. The reception of a Time Exc e e de d message allows the traceroute to ol to infer the presence of the source a ddress at distance h on the path to d . The tr a ceroute tool requires s ome means of matching r eturn pac k ets with the corresp onding prob e pac ke ts to know the correct sequence of IP addresses in the route to d . This is done b y examining the payload of the Time Exc e e d e d pac k et, whic h con tains the b eginning of the prob e pack et. More precisely , an ICMP Time Exc e e d e d message con tains the IP header of t he prob e pac k et, and the ﬁrst eight o ctets that follow the IP header [17, p.5]. If the IP header con tains no options, as is the case for traceroute prob e pac k ets, this amoun ts to the ﬁrst 28 o ctets o f the IP pac ke t. T he eigh t o ctets follo wing the IP header comprise either the en tirety of t he UDP header, the en tirety of the ICMP Echo header (the standa r d four o ctets of the ICMP header plus four o ctets for the Iden tiﬁer and Sequence Number ﬁelds), or part of the TCP header, dep ending on P . If this p o rtion of the prob e pac k et con tains a unique iden tiﬁer, then traceroute can recognize the identiﬁer in the Time Exc e e de d pack et and matc h it to the correspo nding prob e. The default b ehav ior of traceroute consists in setting t he Source P ort v alue of UDP prob es to the running pro cess identi- ﬁer (PID) plus 32,7 68, and the initial Destination P ort v alue to 33,435, and incremen ting it with each prob e sen t. F or ICMP Echo prob es, the Sequence Num b er ﬁeld is incremen ted with eac h prob e sent [1]. F or v arious reasons (suc h as routers dropping prob es with a TTL o f 1 without notifying the source, or routers o n the rev erse path dropping ICMP T i m e Exc e e de d messages), there migh t b e no answ er to a giv en prob e. If, af ter a giv en time in terv al has elapsed, traceroute has not receiv ed an answ er f or a giv en prob e, it stops w aiting for it and outputs a star (‘*’) fo r the corresp onding prob e. 2.2 T r ac er oute and lo ad b alancing Net w ork administrators employ load balancing to enhance reliability and in- crease resource utilization. The main w a y to do so is through the in tra-do main routing proto cols OSPF [18] and IS-IS [19] that supp ort e qual c o st multip ath ( ECMP ). An o p erator o f a m ulti-homed stub net w ork can also use loa d bal- ancing to select whic h of its inte rnet servic e pro viders will receiv e whic h pac k- ets [20]. 2 F or m ore detai ls, see Ma o et al. [16] and r eferen ces within. 4 Routers can spread t heir traﬃc across m ultiple equal-cost paths using a p er- pac k et, per- ﬂow, or p er-destination p olicy [21,2 2]. In p er-ﬂow lo ad b alancing , pac k et header information ascrib es eac h pack et t o a ﬂow , and the router for- w ards all pac k ets b elonging to a give n ﬂow to the same path. This helps t o a v oid pac ke t reordering within a ﬂow . Per-p acket lo ad b alancing mak es no attempt to k eep pack e ts fro m the same ﬂow together, and fo cuses purely on main taining an even loa d o n paths. This migh t b e through ro und- robin assign- men t of pac k ets to paths. The balance cannot b e disturb ed by the presence of out-sized ﬂows. Finally , p er-destination lo ad b alancing could b e seen as a coarse f orm of p er-ﬂo w lo ad balancing, as pack ets are directed as a f unction of their destination IP address. But, as it disre gar ds source inf o rmation, there is no notion of a ﬂo w p er se . As seen from the measuremen t p oint of view, p er-destination load balancing is equiv a len t to classic ro uting, whic h is also p er destination. Concerning p er-ﬂo w load balancing, a natural ﬂow iden tiﬁer is t he ﬁve-tuple of ﬁelds from the IP header and either the TCP or UDP header: Source Ad- dress, D estination Address, Proto col, Source Port, and Destination P ort. W e p erformed some tests on some load-balancing routers from our traces to get an indication of whic h ﬁelds a r e used by routers to determine whether tw o pac k ets b elong to a same ﬂo w or not. W e used TCP , UDP , ICMP , a nd IPSec prob es. W e sen t prob es from o ur lab oratory to diﬀerent destinations that cross the sele cted routers and v aried header ﬁelds t o observ e whic h ones trigg ered load balancing. These tests sho w ed t hat r o uters balance load using v a rious com binations of the ﬁelds of the ﬁv e-tuple, as we ll as three other ﬁelds: the IP T yp e of Service (TOS), and the ICMP Co de and Chec ksum ﬁelds. W e hav e not obtained a nsw ers from routers fo r ICMP prob es of an y t yp e ot her than ICMP Echo , whic h means that w e w ere unable to ascertain whether the ICMP T yp e ﬁeld is also used for p er- ﬂo w lo ad balancing or not. Fig. 2 summarizes the IP , UDP , ICMP Ech o , and TCP header ﬁelds that we observ ed are used b y p er-ﬂow load bala ncers. W e lea v e an exhaustiv e study of whic h pa r t s of the header ﬁelds serv e for load balancing, and in precisely whic h w ays , to future w ork. Finally , whether a router ba la nces load p er-pack et, p er-ﬂow or p er-destination dep ends on the router man ufacturer, the O S v ersion, and ho w the netw ork op- erator conﬁgures it. F or instance, Cisc o and Junip er ro uters can b e conﬁgured to do any of the three t yp es of load balancing [21,23,22]. T r a ceroute, in consequence of its design, systematically sends prob es via dif- feren t paths in the presence o f p er-ﬂo w load balancing. This comes from its manipulation of the con ten ts of the 28 ﬁrst o ctets of the prob es, in order to obtain a unique iden tiﬁer. When sending UDP prob es, it systematically v aries the Destination P ort ﬁeld. When sending ICMP Echo prob es, it v aries the Sequenc e Num b er ﬁeld. Ho w ev er, as explained ab ov e, v arying these ﬁelds 5 TTL = 8 TTL = 7 Possible traceroute outcome: TTL = 6 TTL = 9 Hop 6 Hop 7 Hop 8 Hop 9 Hop 9 Hop 8 Hop 7 Hop 6 2 0 1 0 0 1 1 0 0 1 2 1 1 0 S L E C A B D 0 0 A D L 0 E 1 Fig. 1. Missing routers and links, and false link s . amoun ts to c hanging the ﬂow iden tiﬁer for each prob e. The D estination P ort ﬁeld in the UDP header is used for ﬂo w identiﬁc atio n, and, though the Se- quence Num b er ﬁeld is not dir ectly used in this w ay , v arying this ﬁeld v aries the Chec ksum ﬁeld, whic h is a ﬂow iden tiﬁer. Where there is load balancing, there is no longer a single route from a source to a destination. Classic traceroute is not only unable to uncov er all routes from a s ource to a giv en destination, but it also pro v es unable to iden tify one single route from among man y . It suﬀers fro m tw o systematic problems: it fails to disco v er routers and links, and it may unco v er false links. This is illustrated in Fig. 1. Here, L is a load bala ncer at hop 6 from the traceroute source, S . On the left, w e see the true router top ology from hop 6 to hop 9. Circles represen t routers, and eac h router interface is n umbered. Blac k squares depict prob e pack ets sen t with TTLs 6 through 9. They are sho wn either a b ov e the top ology , if L directs them to router A , or b elo w, if L directs them to router B . On the righ t, w e see the top olo gy that would b e inferred from the routers’ resp onses. Missing routers and links. Because routers B and C send no resp onse, they are not disco ve red, and links suc h as ( L 0 , B 0 ) and ( B 0 , D 0 ) cannot b e inferred. The probability o f missing routers g o es up with the num ber of routers at a giv en hop coun t: for instance, if there are four routers a t a giv en hop, assuming that all prob es hav e an equal probability of rep orting each of the four routers, the pro babilit y of missing at least o ne o f them when sending four prob es is appro ximately equal to 0 .78. Notice also that where there are missing ro uters, there are necessarily missing links as well. Finally , notice that the num ber of routers at a giv en hop count ma y b e quite high: the new er Junip er routers, for instance, p ermit up to sixteen equal-cost paths. F alse links. Independently of whether all routers (or links) ar e discov ered or not, the traceroute to ol lends itself to the inference of false links. In o ur example (Fig. 1), L directs the prob e with initial TTL 7 to A and the one with initial TTL 8 to B , leading to the mistak en inference of a link b et w een A 0 and D 0 . 6 In o ur example, t here is a probabilit y 2 / 2 4 = 0 . 12 5 that a ll prob es to hops 7 and 8 follow an iden tical path. Th us, there is a probability of 0 . 8 7 5 that addresses from routers on diﬀeren t paths will b e rev ealed on subseq uent ho ps, leading to false links imp ossible to diﬀeren tiate fro m true ones in suc h mea- suremen ts. Again, the presence of a high num ber of routers at a giv en hop coun t complicates the problem further b y increasing the proba bility of infer- ring false links. The problem of fa lse links inferred from the app eara nce of m ultiple addres ses on a same hop has b een ac know ledged in some cases, in particular b y Huﬀak er et al. [3] and Spring et al. [4]. Ho w ev er, no systematic study of this phenomenon has b een undertak en, and no real solution has b een prop osed. 2.3 A new tr ac er oute W e now in tro duce Paris tr ac er oute , 3 a new t r a ceroute de signed fo r net w orks with load-balancing routers. Its k ey inno v ation is to control the prob e pac k et header ﬁelds in a manner that mak es all prob es tow ards a destination follow the same path in the presence of p er-ﬂow load balancing. It also mak es it p ossible to distinguish b et w een the presence of p er-ﬂow lo a d balancing and p er-pac ke t load balancing, as we see b elow. Unfortunately , due to the random nature of p er-pack et load balancing, P aris traceroute cannot p erfectly en u- merate all paths in all situations. But it can do considerably b etter than t he classic traceroute, and can ﬂag those instances where there are doubts. Main taining certain header ﬁelds constant is diﬃcult b ecause traceroute needs to matc h resp onse pack ets to their corresp o nding prob e pac k ets, and, a s Fig. 2 sho ws, there is limited space in the prob e pac k et headers in whic h to enable this matc hing (only the ﬁrst 28 o ctets of the pr o b e pac k et are encapsulated in the ICMP Time Exc e e de d resp onse). Whithin this space, it is necessary to enco de a pac k et identiﬁer, and not all ﬁelds can b e used for this purp ose if the pac k ets a r e to b elong to a single ﬂo w. Also, to a v oid am biguity when there are m ultiple instances of the program running on the same mac hine, b ot h traceroute and Paris traceroute enco de a pro cess iden tiﬁer into the prob e. Some header ﬁelds, suc h as the IP V ersion, simply cannot b e altered from their original purp ose, as routers w ould tend to discard the pac k ets a s malformed. Other ﬁelds, suc h as the TTL, Source Addre ss, and Destination Address, nat- urally cannot b e altered. Nor would it b e suitable to enco de iden tiﬁers into the IP options, as pac k ets with IP optio ns are typ ically pro cessed oﬀ the routers’ ‘fast path’ [2 4 ], meaning by diﬀerent co de, and t hus with p ossibly diﬀerent forw arding rules than the normal pac k ets whose paths traceroute is supposed 3 Av ail able at http://www.p aris- traceroute.net/ 7 to trace. F inally , as w e hav e men tioned, the IP TOS ﬁeld a nd the ﬁrst four o ctets of the transp ort la y er header are oﬀ b ounds as they are used for load balancing. P aris traceroute uses the sev en th and eigh t o ctets of the transp ort la y er header as the pack et iden tiﬁer. In UDP prob es, this is the Chec ksum ﬁeld. How ev er, simply setting the c hec ksum v alue without regard t o pac ke t con ten ts w ould create ill-formed prob e pack ets liable to b e discarded as corrupt. T o obtain the desired c hec ksum, Paris traceroute manipulates the UDP pa yload. F or ICMP Echo prob es, P aris tra ceroute uses the Sequence Num b er ﬁeld, as do es classic traceroute. Ho we ve r, in classic t raceroute this has the eﬀect of v arying the ICMP Chec ksum ﬁeld, whic h is used for load balancing. Paris traceroute main tains a constant ICMP Chec ksum b y changing the ICMP Iden- tiﬁer ﬁeld to oﬀset the c hange in the Sequence Number ﬁeld. P aris traceroute also sends TCP probes, unlike classic traceroute, but lik e T o ren’s v ar ia n t tcptraceroute [13]. T o identify TCP prob es, P aris traceroute uses the Sequence Number ﬁeld (instead of the IP Iden tiﬁcation ﬁeld used b y tcptraceroute). No o ther manipulations are necessary in order to main tain the ﬁrst four o ctets o f the header ﬁeld constant. T o enco de the pro cess iden tiﬁer, P aris traceroute uses the IP Identiﬁc atio n ﬁeld, whereas classic traceroute uses the Source P ort for UDP prob es, and the Iden tiﬁer f or ICMP Ec ho prob es. Tcptraceroute do es no t enco de a pro cess iden tiﬁer into its prob e pa ck ets. The simple fact of main taining a constan t ﬁv e-tuple is not orig inal to P aris traceroute, as tcptraceroute already do es t his for the TCP prob es that it sends: in order to more easily t ra v erse ﬁre walls, tcptraceroute b y de fault sets probes’ Destination Port ﬁeld to 80, emu lating we b traﬃc. No prior w ork has, ho we ve r, examined the eﬀect of this c hoice with resp ect to lo a d balancing. In addition to ﬁxing the ﬂow iden tiﬁer problem, Paris traceroute pro vides additional information useful for recognizing certain other traceroute mea- suremen t artifa cts. T he pr ob e TT L is t he TTL that is found in the IP header of the prob e pac k et encapsulated in the ICMP Time Exc e e de d response. This v alue corresp onds to the prob e’s TTL when the router receiv ed it and decided to discard it. Under normal traceroute b ehavior, this v alue is o ne. A v alue other than o ne rev eals an error. The r esp onse TTL is the TTL of the T ime Exc e e de d respo nse itself. This v alue, av ailable also through classic tra ceroute, helps in inferring the length of the return path. Finally , the IP ID is the Iden- tiﬁcation ﬁeld from the IP he ader of the Time Exc e e de d resp onse. Th is ﬁeld is set by the ro uter with the v alue o f an in ternal 16-bit coun ter that is usually incremen ted fo r eac h pac ke t sen t. The IP ID can help identify the m ultiple in terfaces of a single router, as describ ed in the R o cketfuel w ork [4], or un- 8 Used for per−flow load balancing * Varied by Paris traceroute + Varied by tcptraceroute Key Not encapsulated in ICMP Time Exceeded packets Source Address Destination Address TTL Protocol Header Checksum Identification (+) Flags Fragment Offset Version IHL TOS Total Length IP UDP ICMP Echo TCP Destination Port (#) Length Checksum (#,*) Identifier (*) Sequence Number (#,*) Type Code Source Port Destination Port Acknowledgment Number Data Offset ECN Window Checksum Urgent Pointer Options and Padding Sequence Number (*) Source Port Resvd. Options and Padding Control Bits Checksum (#) # Varied by classic traceroute Fig. 2. The role s pla ye d by pac ket h eader ﬁelds . co v er diﬀeren t r o uters and hosts hidden b ehind a ﬁrew all or a NA T b o x, as described b y Bello vin [25]. Finally , P aris traceroute ma y use diﬀeren t strategies f or sending prob es. Pack- etbyPacket b ehav es lik e the classic tra ceroute, i.e., sends a pro b e, w aits for an answ er or a timeout, sends the follo wing prob e, and so on. HopByHop sends all the prob es for a given hop with a conﬁgurable delay b et w een prob es (the default v alue o f this dela y is 50 ms), waits for answ ers or timeouts, and then rep eats the same pro cedure for the next hop. HopByHop is faster than P ack et- ByP ac k et. Conc urr ent sends all the pro b es for all the hops with a conﬁgurable dela y (default 50 ms) b etw een prob es. This algorithm is signiﬁcan tly faster than the previous t w o. T o use Concurren t, how ev er, one must k now t he n um- b er of hops needed to reac h the destination. Sc out sends one prob e with a v ery high TTL to the destination. If the destination resp onds, then it estimates the n um b er of hops needed to reac h the destination and uses Concurrent. Other- wise, this strategy cannot b e use d. Scout is similar to the r ac er oute techniq ue prop osed by Mo ors [9]. P aris tra ceroute also allo ws the user to customize the minim um and maxim um TTL v alues, in order to fo cus the measuremen t o n a par t of the path, and th us further sp eed up the measuremen ts. 9 3 Measuremen t setup This section explains ho w we conducted side-b y-side measu remen ts with clas- sic traceroute and P aris tra ceroute, in or der to study traceroute measuremen t artifacts. Our measuremen t source w as lo cated at the LIP6 lab oratory of the Univ ersit ´ e Pierre et Marie Curie, in P aris, F rance. The unive rsity has only one connection to the in ternet via the F rench academic bac kb one, Renater. Our destination list consisted o f 5,000 IPv4 addresses chose n at ra ndo m, with- out duplicates, that answ ered to a ping prob e a t the time o f the creation of the list. F ollo wing the lead of Xia et a l. [26], w e only considered pingable addresses so as to av oid the artiﬁcial inﬂation o f traceroute measuremen t a r t ifacts in our results that w ould come from tracing to w ards un used IP addresses. F or a p eriod of 74 days b et w een June and August 2006, we p erformed 1,465 rounds of measuremen ts by using classic traceroute and P aris traceroute to prob e from our source to a ll destinations. This yielded the data set we use throughout this pap er. This data set serv es as a case study , to sho w how o ne can identify , explain, and remov e traceroute measuremen t artifacts with Paris traceroute. It is not in tended to b e represen tativ e of the statistics of traceroute measureme nt a r- tifacts on the in ternet in general. W e conducted sev eral other measureme nts with similar setups and obtained results consisten t with the ones w e presen t b elo w. Our results should therefore b e considered as represen tativ e of what w e observ e fr om this sour c e and under these me asur ement c onditions , but statis- tics obta ined from other v antage p o ints, or tow ards o ther destination sets, ma y v ary . Obtaining a repre sen tative view of what can b e seen on av erage on the interne t is clearly an in teresting question, but it is out of the s cop e of this pap er. Eac h round of measuremen t was conducted in the follow ing w ay . W e launc hed 32 pro cesses in parallel, that each prob ed 1 / 32 nd of the destination list. Eac h pro cess selected, in turn, eac h destination d from its po rtion o f the list, and traced t w o routes to d , ﬁrst using P aris traceroute, then using an instance of classic traceroute (NetBSD version 1.4a 5). F or b oth P aris tra ceroute a nd classic traceroute, w e sen t a single UDP prob e for each hop, using the P ac k et- ByP ac k et strategy: waiting f or a reply or a timeout fro m hop h b efore sending the prob e for hop h + 1. 4 The c hosen timeout was 2 seconds. P aris tr a ceroute k ept the ﬁv e-tuple of its prob es constan t during e ach ins tance of probing to a giv en destination, and selected Source and Destination P ort v alues randomly 4 This is the only p ossible strategy for classic traceroute. 10 from the range [1 0 ,000, 60,000]. Concerning classic t r a ceroute, w e kep t the default b ehavior. The probing tow ards a g iven destination terminated when either t he destination responded, or a n ICMP message other than Ti m e Ex- c e e de d w as receiv ed. Moreo v er, P aris traceroute a lso stopp ed when TTL 36 w as reac hed, or when eight conse cutiv e stars w ere seen, whiche ve r came ﬁrst. Classic traceroute w as set to stop if it reached a TTL of three greater than the last hop at whic h the previous run of Paris traceroute r eceiv ed an answe r. Finally , w e a lw a ys set the minimal TTL to 2 to skip the routers inside the univ ersit y net w ork. One round of measuremen ts t o all destinations to ok approx imately one ho ur and ﬁfteen minutes , a t the rate of approximately 27 .3 seconds for b oth a P aris traceroute and a classic traceroute to a giv en destination. F or the 241 million responses that con tain v alid IP Source Address v alues, w e mapp ed the address to an AS n um b er using Mao et al.’s techn ique [27]. Our data set co v ers 1,498 diﬀeren t ASes, which corresp onds to six p ercen t of the ASes in the in ternet t o day . Our data set cov ers all nine tier-1 ISP netw orks and 7 5 of the one hundred to p-20 ASes of eac h region according to APNIC’s w eekly routing table rep o rt. 5 Stars mostly a pp eared at the end of measured routes (when a destination do es not answe r, our measuremen t metho d induces man y stars at the end o f the corresp onding measured ro ute), with just 7 . 7 million app earing in the midst of respo nses. W e searc hed our measuremen ts for in v alid IP addresses, i.e., addresses that should not b e giv en to any host on the public in ternet [28]. W e found 11 in v a lid IP a ddresses that accoun t for 57 thousand resp onses. None o f these in v alid addresse s app ear in the structures w e study in the next section, and therefore they probably play little role in our observ a t io ns. Finally , w e deﬁne a me asur e d r oute to b e the output of a giv en classic tracer- oute or P aris traceroute instance. F ormally , a measured ro ute is an ℓ -tuple r = ( r 0 , · · · , r ℓ ) where r 0 is t he source address, and, for eac h i , 1 6 i 6 ℓ, r i stands either for the IP address receiv ed when probing with TTL i , o r for a star if none w as receiv ed. The in teger ℓ is called the length of the measured route. W e call an y tuple of the form ( r i , r i +1 , . . . , r i + k ) a subr oute of r of length k . 5 APNIC automatically generates rep orts describing the state of in ternet routing tables. It ranks ASes for eac h of ﬁve world r egions according to the num b er of net wo rks announced. 11 4 Structures under study W e fo cus our observ ations on three particular structures that a pp ear in man y measured routes: w e call t hem lo ops, cycles, and diamo nds. In this se ction w e describe these structures, and presen t some basic statistics ab out their frequency of app earance in our traceroute data set. Sec. 4.1 discusses lo ops, Sec. 4 .2 discusses cycles, and Sec. 4.3 discusses diamonds. In subsequen t sec- tions, we giv e p ossible causes and e xplanations for these structures, a s w ell a s metho ds for distinguishing b etw een the diﬀeren t causes. 4.1 L o ops In some measured routes, the same IP address app ears t wice or more in a r o w: w e call this a lo op . In the norma l course o f routing, a router do es not forw ard a pac k et bac k to the incoming interface. Hence, lo ops are most lik ely an artif a ct of the measuremen t itself. F ormally , a lo op is observ ed on IP address r i with destination d if there is a t least one measured route tow ards d containing · · · , r i , r i +1 , · · · with r i +1 = r i . The term ‘IP address’ implies that r i is not a star. Load balancing can cause lo ops that app ear sp oradically when rep eat edly tracing from a source to the same destination; this is sho wn in Fig. 3. Thes e lo ops can o ccur in t he presence of a loa d bala nced route with at least t w o paths ha ving a length diﬀerence equal to one. TTL = 6 TTL = 8 TTL = 7 TTL = 9 What we see: Hop 6 Hop 7 Hop 8 Hop 9 Hop 8 Hop 7 Hop 6 1 2 0 0 1 1 0 0 1 0 1 2 0 0 0 S L A B C E A L E Fig. 3. Lo op c aused by load bala n cing. W e call eac h observ ation of a measured r o ute a s describ ed ab o v e a lo op in- stanc e , and we deﬁne this lo op’s signatur e as t he pa ir ( r i , d ). F or instance, if w e ha v e four measured rout es ( . . . , a, a, b, . . . , d 1 ), ( . . . , a, a, b, . . . , d 2 ), ( . . . , a, a, b, . . . , d 2 ) and ( . . . , a, a, c, . . . , d 2 ), then w e hav e four lo op instances (one fo r eac h measured route), and t w o lo ops signatures (( a, d 1 ) a nd ( a, d 2 )). D ep ending on the con text, w e will fo cus on statistics in v olving either lo op signatures or lo op instances. Ho w ev er, when the con text is clear or when suc h distinction is ir- relev ant, w e will simply use the term “lo op”. Moreo v er, w e say that a measured route r contains a n -lo op instance, with 12 n > 1, on IP address r when r w as observ ed on exactly n + 1 consecutiv e p ositions on r ( n is the num ber of times r is observ ed at t w o consecutiv e p ositions). W e call n the l eng th o f the lo op instance. The shortest lo o ps are of length 1. W e extend this deﬁnition to lo op signatures: the length of a lo op signature is the maximal length of a ll its instances. Note that w e use the term “lo op” a s mean t commonly in g raph theory . This is diﬀere nt from a for w arding loo p, whic h t ypically in v olv es sev eral address es. F orw arding lo ops are b est describ ed as cycles, w hich are discussed in Sec. 4.2 . Using the ab o v e deﬁnitions, en umerating lo ops is straig h tforward: w e simply scan all the measured ro utes and lo ok for instances of IP addresses that app ear at least twic e consecutiv ely . T o p erform statistics, we maintain a list of all lo op signatures encountere d, and for eac h signature ( r, d ) w e k eep a record of the relev ant information: the n um b er of instances, the maximal length, the n um b er of times the IP address r has b een observ ed on measured routes tow ards d (whether a lo o p has b een o bserv ed or not), and so on. Lo ops app ear to b e surprisingly common: o v erall, 7 . 51% of the IP addresses detected in our experimen t w ere in at least one lo op instance, and more than 4 . 35% of the measured routes contained a lo o p. Moreov er, when probing re- p eatedly , sa y N times to the same destination, the probabilit y t hat at least one of the measured routes contained a lo op increased with time; during our measuremen ts, w e w ere able to observ e lo ops tow ards more than 24 . 6% of the destinations. This is due to the high heterogeneit y of lo ops: while some of them seem to b e p ersistent and app eared on almost eve ry measured route to w ards their destination, many others are o c c asional and app eared o nly once, or a couple of times. M easuring for a lo nger p erio d of time therefore increased the probabilit y of observing suc h o ccasional loops, within the duration of our exp eriments. 6 W e a lso observ ed an intermediate b eha vior: systematic lo ops. W e sa y that a lo op ( r, d ) is sys tematic if, whenev er the IP address r app eared in a measured r o ute to w ards d , this o ccurrence w as part of a lo op. In other w ords, eve ry time w e sa w r , w e saw the lo op. The diﬀerence b et w een systematic a nd p ersisten t lo ops is that systematic lo ops don’t alw ay s sho w up on measured ro utes to a given destination, and may actually b e exceptional. T o examine this furt her, we deﬁned t w o distinct c har- acteristics. Consider a lo op ( r, d ) on address r on a meas ured route to w ards a destination d : (1) Its a p p e ar anc e fr e quency is the probability that a measured route tow ards 6 This p robabilit y w ould, ho w eve r, most lik ely stop growing giv en suﬃcien tly long measuremen ts, as we cannot exp ect to observ e lo ops on measured routes to ev ery single destinati on. 13 d con tained the lo op. (2) Its c o nditional app e ar anc e fr e quency is the probability that a measured route tow ards d that c ontaine d r also contained the lo op. 0 0.1 0.2 0.3 0.4 0.5 0.6 0 0.2 0.4 0.6 0.8 1 Proportion of loops Probability of appearance Appearance Cond. app. Fig. 4. Dark bars: app earance frequency of th e lo ops. Ligh t bars: conditional ap- p earance frequency of the lo ops. Fig. 4 show s the distribution of these t w o c haracteristics ov er the lo op signa- tures w e observ ed. Notice that the proportio n of p ersisten t loops (app earance frequency close to 1) seems to b e v ery small, m uch smaller than the prop ortion of systematic lo o ps (conditional app earance frequency close t o 1). Ho w ev er, since these lo ops were observ ed more of ten than the o thers, their prop ortion as instanc es w as m uch more signiﬁcan t (7 . 45% of all instances). Fig. 5 shows (left) the l e n gths o f the n - lo ops w e observ ed and their distribu- tions among lo op signatures (left) and lo op instances (right). This chart is logarithmic, since large n -lo ops (i.e., with n > 1 ) are v ery uncommon com- pared to 1- lo ops. How ev e r, w e observ ed sev eral p ersis tent lar g e n - lo ops, whic h indicates that they shouldn’t b e considered as marginal eve nts. 1e-06 1e-05 0.0001 0.001 0.01 0.1 1 3 6 9 12 15 18 21 24 Freq. among signatures Length of the loops 1e-06 1e-05 0.0001 0.001 0.01 0.1 1 3 6 9 12 15 18 21 24 Freq. among instances Length of the loops Fig. 5. Distribution of the length of lo op signatures (left) and instances (right ). Both length and app earance statistics conﬁrm that sev eral ‘ﬂav ors’ of lo ops exist. This p oints to the unlik eliho o d that a ll lo ops are generated by a unique mec hanism, and suggests c haracterization o f the lo ops into diﬀerent categories, eac h o f them resulting fr o m a diﬀeren t cause. 14 4.2 Cycles F ormally , a measured route r is said to b e cyclic on an IP address r , or r -cyclic, if it con tains r at least twice , at nonc onse cutive lo cations, i.e., sep- arated b y a t least one IP address r ′ distinct from r . The term ‘IP address’ implies that neither r nor r ′ are stars. This distinction is to mak e sure we don’t misin terpret p ossible n - lo ops as cycles. F or instance, a measured r oute ( . . . , b, c, d, e, c, f , . . . ) is cyclic on c , but the measured route ( . . . , b, c, ∗ , ∗ , c, f , . . . ) is not, since the sequence c, ∗ , ∗ , c migh t w ell b e a 3-lo op. Load balancing can cause cycles, in the same w a y as lo ops; see Sec. 4.1 and Fig. 3. Cycles can o ccur when there is loa d balancing betw een routes having a length diﬀerence larger than one. As for lo ops, w e use the term cycle instanc e for an y o ccurrenc e of a cycle on a measured route, a nd deﬁne a cycle signatur e as a pa ir ( r, d ) of an IP ad- dress and a destination such tha t at le ast one of the measured routes tow ards d is cyclic on r . Most of the cycles w e observ ed o ccurred at more than one lo cation on the measured routes, and sometimes they eve n ov erlap with eac h other, making it hard to deﬁne prop erties of a cycle as if it we re a single, w ell-deﬁned ob ject. T o b e able to pro vide some statistics, w e consider t w o prop erties. The length of a cycle signature ( r , d ) is the shortest distance that separates t w o instances of r in all measured routes tow ards d that are r - cyclic. The sp an of a cycle signature ( r , d ) is the gr e atest dis tance that separates tw o instances of r in a ll measured routes tow ards d that are r - cyclic. These prop- erties capture resp ectiv ely the minimal and ma ximal length of a round-trip from r to r observ ed on a measured route tow ards destination d . F or instance, if w e hav e three measured routes ( . . . , r, a, b, r, . . . , d 1 ), ( . . . , r , a, b, r, . . . , d 2 ) and ( . . . , r , c, d, e, r, . . . , d 2 ), then the cycle signature ( r, d 1 ) has b o t h length and span equal to 3 , whereas the cycle signature ( r, d 2 ) has a length of 3 and a span of 4 . An in teresting species of cycles is the p erio di c cycles , whic h are encoun tered in measured routes like ( . . . , a, b, c, b, c, . . . ). A measured route is k -p erio dic al ly cyclic on IP addresses r 1 , r 2 , · · · , r k when this sequenc e of IP addresses app ear at least t wice, consecutiv ely , and in the same order, on the measured route. F ormally , a p eriodic cycle signature is a pair (( r 1 , · · · , r k ) , d ) suc h that at least one measured route to w ards d is k -p erio dically cyclic on r 1 , · · · , r k . As for lo ops, en umerating cycles consists in scanning ev ery measured route, detecting the cycle instances, and aggregating informa t io n ab out ev ery cycle signature encoun tered. Cycles ar e less common than lo ops in our dat a set: they app eared on o nly 15 0 . 60% of the measured routes. On the other hand, they app eared o n a broad range of a ddresses: we observ ed cyclic measured routes tow ards 17 . 5 % of the destinations, a nd 4 . 7 3 % of the IP addresses disco v ered during o ur exp erimen t app ear in at least one cycle signature. Fig. 6 presen ts app earance statistics similar to the ones alr eady described for lo ops, using a similar terminology . T o a v oid redundancies, w e in vite the reader to consult Sec. 4.1 for deﬁnitions. Persis tent cycles app ear to b e v ery 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.2 0.4 0.6 0.8 1 Proportion of cycles Probability of appearance Appearance Cond. app. Fig. 6. Dark bars: app earance frequency of cycles. Light bars : conditional app earance frequency of the cycle s. rare; they are actually in visible on F ig. 6. W e did detect 2 p ersiste nt cycles o v er the 5,674 cycle signatures w e observ ed, and they account for 3 . 93% of the cycle instances. Sys tematic cycles w ere more common (see the p eak in Fig. 6 for a conditio na l app earance frequency of 1), and repres ente d 8 . 95% of the cycle instances . They still ha v e a v ery low app earance fr equency , me aning that in spite of their systematic b eha vior, their o ccurrence remained infr equen t in our measuremen ts. Ove rall, it seems that cycles, unlik e lo ops, are mostly o c c asional ev en ts. This mak es them more diﬃcult to trac k. Fig. 7 sho ws the distribution o f the length and span of the c ycles w e observ ed (as fo r lo ops, w e both use the signature- and instance-based statistics). These statistics illustrate the v ariet y of cycles that can b e observ ed. F rom the distribution o f the length of cy cles, it is clear that cycle s of length 2 w ere predominan t, esp ecially for cycle instances. Ho w ev er, the cycles ha ving a span of 2 w ere not as common, and they w ere ev en outnum bered b y cycles of span 3 in terms of instances, whic h indicates that no t all cycle s of length 2 also ha v e a span equal to 2. This is related to the observ ation that, for higher span v alues , cycles with an ev en span are muc h more likely to o ccur than those with o dd span. These observ ations can b e explained in terms of the p erio dic cycles w e introduced earlier: a p erio dic cycle o f p erio d 2 will ha v e a length equal to 2, and may ha v e an y ev en sp an b et w een 2 and the maxim um num ber of hops allow ed in our measuremen ts. This, plus the fact that cycles of extreme length also o ccurred regula r ly (ac- cording to the length distribution), also suggests the existence of a wide v ariet y of phenomena as p ossible causes for cycles. 16 1e-05 0.0001 0.001 0.01 0.1 1 0 5 10 15 20 25 Freq. among signatures Length of the cycles 1e-05 1e-04 0.001 0.01 0.1 1 0 5 10 15 20 25 Freq. among instances Length of the cycles 1e-05 0.0001 0.001 0.01 0.1 1 0 5 10 15 20 25 30 35 Freq. among signatures Span of the cycles 1e-05 1e-04 0.001 0.01 0.1 1 0 5 10 15 20 25 30 35 Freq. among instances Span of the cycles Fig. 7. Distribution of the length (top) and span (b ottom) of t h e cycles signatures (left) and instances (righ t). 4.3 Diamonds Lo ops and cycles are structures observ ed on single measured routes. Other t yp es of structures in traceroute measuremen ts a pp ear only when m ultiple measured routes are considere d together (e.g., when constructing maps of the net w ork). A ty pical feature that w e observ e in this w a y is what w e call a diamond. Giv en a set S of measured ro utes, a diamond is a pair ( h, t ) of IP a ddresses for whic h the num ber k of distinct IP a ddresses r i suc h that there exists a measured route in S of the form . . . , h, r i , t, . . . is at least 2. The term ‘IP address’ implies that neither h , t , nor any of the r i are stars. W e call h the he ad of the diamond, and t its tail . The c or e of the diamond is the set of addresses { r 1 , . . . , r k } , and the diamond’s size is k . Load balancing may induce man y diamonds. Fig. 8 presen ts a typical suc h case. Possible traceroute outcome: Hop 6 Hop 7 Hop 8 Hop 9 3 0 1 0 0 1 0 3 1 0 0 1 1 2 1 0 1 0 1 2 0 0 0 0 0 0 0 L G D A B E C F A B E C L D G Fig. 8. An example of sev eral diamonds caused b y loa d balancing. F or clarity , we omit t h e prob e p ac k ets. 17 Whic h diamonds w e see dep ends up on whic h set S of measured routes is considered. W e will see in Sec. 5.3 that the distinctions b et w een diﬀeren t t yp es of diamonds aﬀo r d in teresting observ ations concerning loa d balancing. A diamond ( h, t ) that emerges fro m only the measured routes to w ards a single destination d is called a d -diamond . If there exis ts at least one destination d for whic h ( h, t ) is a d -dia mo nd, then w e call ( h, t ) a one-destination diamond . W e deﬁne the size of a one-destination diamond ( h, t ) to b e the maximum of the sizes of all d -diamonds ( h, t ). A diamond ( h, t ) that emerges fr o m the entiret y of measured ro utes in our data set is called a glob al diamond . All one-destination diamonds are also global diamonds. ... 0 0 ... ... ... 0 0 1 0 0 0 0 0 0 0 0 0 0 1 2 2 0 d L A B C D E F G H I J K d d d N O M Fig. 9. Examples of global diamonds, d -diamonds , and on e-destination diamonds. Fig. 9 presen ts examples of d -diamonds, one-destination diamonds, and global diamonds. W e see measured routes tow ards tw o distinct destinations d 1 and d 2 . A 0 , . . . , O 0 are addresses o bserv ed on these measured ro utes, and a solid (resp ectiv ely , dashed) arrow b et w een t w o addresses indicates that they are link ed in a measured route tow ards d 1 (resp ectiv ely , d 2 ). ( G 0 , L 0 ) is a d 1 - diamond of size 3, and a d 2 -diamond of size 2. Hence ( G 0 , L 0 ) is also a one- destination diamond, and its size is 3, whic h is the size of the lar gest d -diamond ( G 0 , L 0 ). ( G 0 , M 0 ), ( J 0 , O 0 ) and ( K 0 , O 0 ) are d 2 -diamonds of size 2 and also one-destination diamonds o f size 2. Note that ( A 0 , D 0 ) is no t a d -diamond for an y d , and therefore not a one-destination diamond. ( A 0 , D 0 ), ( G 0 , L 0 ), ( G 0 , M 0 ), ( J 0 , O 0 ) and ( K 0 , O 0 ) are global diamonds. Diamonds are detected in the followin g manner: w e scan ev ery measured route, and main tain, for eac h p ossible triple ( h, t, d ) o f t wo IP a ddresses and a des- tination, the set A d ( h, t ) of all IP addresses seen b et we en h and t on mea- sured ro utes tow ards d . W e then compute the set A all ( h, t ) of IP addresses seen b et we en h and t on al l measured ro ut es b y merging all the A d ( h, t ). F or a n y destination d , the d - diamonds are then the pairs ( h, t ) suc h that | A d ( h, t ) | > 2. One-destination diamonds are the pairs ( h, t ) suc h that there exists a d suc h that | A d ( h, t ) | > 2, and global diamonds are the pairs ( h, t ) suc h that | A all ( h, t ) | > 2. 18 1e-05 1e-04 0.001 0.01 0.1 1 0 10 20 30 40 50 60 70 80 Frequency Size of the global diamonds Fig. 10 . Distribution of the size of the global diamonds. Ov erall, w e observ ed 28,231 global diamonds in our traceroute data set. Fig. 1 0 presen ts their size distribution. They in v olv ed a v ery large fraction of the observ ed IP addresses: ov erall 44 . 6% of the a ddresses we re inv olv ed in global diamonds. Moreo v er, global diamonds w ere not separate en tities but w ere highly in terlo ck ed with eac h other: 16 . 4% of all addresses w ere in the head of one or mo r e global diamonds, 30 . 7% in the core, and 27 . 1% in the ta il. The sum of these prop o rtions is m uc h larger than 44 . 6%, whic h indicates that a large n um b er of addresses b elonged to sev eral globa l diamonds. One-destination diamonds and d -diamonds w ere a lso quite frequen t in our measuremen ts: there w ere 95,93 6 d - diamonds, and 2 4,326 one-destination dia- monds. Among all destinations d , 90 . 2% led to the observ atio n of at least one d -diamond. The av erage num ber of distinct d -diamonds observ ed with these destinations w as 21 . 3. 1e-05 1e-04 0.001 0.01 0.1 1 0 2 4 6 8 10 12 14 16 18 20 22 Frequency Size of the diamonds 1e-05 1e-04 0.001 0.01 0.1 1 0 2 4 6 8 10 12 14 16 18 20 22 Frequency Size of the d-diamonds Fig. 11. Distribution of the size of one-destination diamonds (left) and d -diamonds (righ t). Fig. 1 1 presen ts the distribution of the size of d -diamonds and one-destination diamonds. W e observ e that there were far few er large d -diamonds than global diamonds. This, tog ether with the f a ct that there a re more globa l diamonds than one-destination diamonds, indicates that measured routes tow ards dif- feren t destinations comb ined with eac h ot her, creating global diamonds where there w ere no one-destination diamonds (as illustrated in Fig. 9), and in- creasing the size of existing one-destination diamonds to form large glo bal diamonds. 19 5 Measuremen t artifacts due to load balancing W e now turn to the study of the diﬀerences o bserv ed when probing with P aris traceroute rather t han with traceroute, for eac h of the three types of structures w e study . 5.1 L o ops W e compare the measuremen ts obtained with classic t r a ceroute and the ones obtained with Paris traceroute, the latter av oiding p er-ﬂow load balancing. Of the lo op signatures, 89 . 8% disapp ear; ho w ev er, new lo o p signatures also app ear, whic h represen t a pproximately 2 . 73% of the initial total. F or lo op instances, the statistics are 79 . 9% a nd 0 . 21%. This sho ws that load bala ncing is lik ely the primary cause of lo ops in our observ a tions. The fact that some lo ops are observ ed in our data set with P aris traceroute but not with classic traceroute most lik ely comes fro m the fact that some lo o ps are rarely observ ed, as explained in Sec. 4.1. It is therefore natural t hat suc h lo ops migh t only b e observ ed with one me asuremen t to o l and not the other. 7 These observ ations also hold for cycles and diamonds. W e in v estigate this matter further by lo o king at the c haracteristics of the lo ops remo v ed by P aris traceroute. F ig. 12 sho ws the conditiona l app earance frequencies (see Sec. 4.1) of lo ops found in the tra ceroute and Paris traceroute data sets. One can see that a lmost all lo ops that w ere neither systematic nor truly rare, i.e., lo ops appearing sp or adically , but somewhat regularly – whic h is the kind of b eha vior one w ould exp ect from loops caused by load balancing – are remov ed. A t the same time, all p ersiste nt lo ops that app eared with traceroute w ere also presen t when using P aris traceroute, whic h was exp ected, since p ersisten t lo o ps ar e unlik ely to b e caused b y load balancing. This will b e discussed further in Sec. 6. W e also observ e that 89 . 3% of the non-p ersisten t systematic lo ops disapp ear. W e ha v e seen how a load-bala nced route with a pair o f path lengths that diﬀer b y 1 can lead to the observ ation of a lo op. W e now h yp othesize a router imple- men tation of p er-ﬂow load balancing that w ould cause suc h a lo op to b e b oth systematic and non-p ersisten t. W e use Fig. 3 as an illustrat io n. If the router L we re to forw ard prob e pac k ets in a round- r o bin fashion, alternativ ely to B and to A , then our experiments would rev eal only t wo measured ro utes out of the man y that are p ossible: one with the s ubroute ( L 0 , B 0 , E 0 , E 0 ) and one 7 This also implies that a small fraction of the lo op s that disapp eared when consid- ering Paris traceroute rather than classic traceroute in our data set may ha ve not b een caused by load bala ncing but b y rare ev ents. 20 0 0.1 0.2 0.3 0.4 0.5 0.6 0 0.2 0.4 0.6 0.8 1 Proportion of loops Conditional prob. of appearance 0 0.1 0.2 0.3 0.4 0.5 0.6 0 0.2 0.4 0.6 0.8 1 Proportion of loops Conditional prob. of appearance Fig. 12. Conditional ap p earance fr equencies of loops observ ed in our data set with classic trace route (left) and Pa ris traceroute (righ t). with ( L 0 , A 0 , C 0 , F 0 ), where F 0 is the next hop b ey ond E 0 . The ﬁrst subroute con tains a lo op t ha t is systematic, b ecause it app ears whenev er E 0 app ears, and that is non-p ersisten t, b ecause it only app ears on a p ortion (in this case, 50%) of the routes to the destination. What w ould accoun t for suc h round-robin fo r warding a t a p er- ﬂo w load bal- ancer? Recall that p er- ﬂow loa d balancing a ssigns pack ets to outgoing in ter- faces a s a function o f the 5-tuple, and that classic traceroute incremen ts the Destination P ort by 1 with eac h sub sequen t prob e, lea ving the rest of the 5-tuple unc hanged. W e obtain the h yp othesized b eha vior if the function em- plo y ed b y the router cycles thro ugh eac h in terface in turn with each incremen t in the Destination P ort. These observ at io ns conﬁrm that load ba la ncing, b e it ﬂo w-based or pac k et- based, is an imp o rtan t source of lo ops. Ho w ev er, we will see in Sec. 6 that some lo o ps, whic h represen t a signiﬁcan t part of the total (7 . 45%), are not due to loa d bala ncing. This is true in particular for p ersisten t lo ops. 5.2 Cycles As for lo ops, w e compare the cycles observ ed with classic tra ceroute and those obtained with Paris traceroute. W e observ e t ha t 79 . 5 % of cycle signatures dis- app ear under Paris traceroute, and that new signatures, represen ting 11 . 0% of the initia l total, app ear. F or cycle instances, these n um b ers are 3 9 . 7% and 1 . 32%. The diagnosis here still is that load balancing is an import an t cause, and most probably the prominen t one, for cycles. The low num ber of cycles caused b y load balancing can b e attributed to the unlik eliho o d o f load bal- ancing across paths having a length diﬀerence o f tw o or more. As for lo ops, w e also inv estigate the eﬀect of Paris traceroute on the app ear- ance frequency of cycles. Fig. 13 shows that cycles that are neithe r syste matic nor v ery rare tend to disapp ear almo st completely . This leads to the same conclusion: lo ad balancing is an imp ortan t cause of the app earance of cycles, 21 0 0.1 0.2 0.3 0.4 0.5 0.6 0 0.2 0.4 0.6 0.8 1 Proportion of cycles Conditional prob. of appearance 0 0.1 0.2 0.3 0.4 0.5 0.6 0 0.2 0.4 0.6 0.8 1 Proportion of cycles Conditional prob. of appearance Fig. 13. Conditional app earance frequencies of cycles observ ed in our d ata set w ith classic trace route (left) and Pa ris traceroute (righ t). and they are signiﬁcantly reduced b y the use of Paris traceroute. 5.3 Diamonds W e no w compare the diamonds observ ed wh en using P aris tr aceroute t o t ho se observ ed with classic traceroute. W e observ e far few er global diamonds with Paris traceroute: 5 2 . 6% of the diamonds o bserv ed with classic tra ceroute disapp ear; con v ersely , new g lobal diamonds, represen ting 3 . 26% of the total observ ed with traceroute, app ear. Fig. 14 prese nts the distribution of the size of glo bal diamonds observ ed with P aris traceroute. 1e-05 1e-04 0.001 0.01 0.1 1 0 10 20 30 40 50 60 70 80 Frequency Size of the global diamonds 1e-05 1e-04 0.001 0.01 0.1 1 0 10 20 Frequency Size of the global diamonds Fig. 14. Di strib ution of the s ize of the global diamonds seen with classic trace route (left) and Pa ris traceroute (righ t). W e observ e that, not o nly are there far few er glo ba l diamonds when using P aris traceroute, but their sizes are also greatly reduced. Moreov er, they are less in tert wined: 11 . 3% of t he IP addresses b elong to at least one global diamond’s head, 2 4 . 8% to a core, and 21% to a tail. The sum of t hese prop o rtions is 57.1, with o v erall 39 . 5% o f the addresses b elonging to a global dia mo nd (compared to 16 . 4% o f a ddresses in a global diamond’s head, 30 . 7% to a core, 2 7 . 1% to a tail, the sum o f these prop ortions b eing 74 . 2, with 4 4 . 6% of a ll addresses b elonging to a global diamond, for classic traceroute). The n umber, size and complexity of d -diamo nds and one-destination diamonds are also gr eat ly reduced: 57 . 1% of the d -diamonds and 56 . 9% of the one- 22 destination diamonds disapp ear; con ve rsely , 2 . 18% and 3 . 26% of the total of d -diamonds a nd one-destination diamonds observ ed with tra ceroute app ear. In addition, among all destinations d , 89 . 6% lead to the observ ation of d - diamonds (compared to 90 . 2% for classic traceroute), and eac h suc h desti- nation leads on a v erage to the observ ation of 9 . 7 d - dia monds (compared to 21 . 3 for class ic tra ceroute). Fig. 15 pres ents the distribution of the size o f d -diamonds observ ed with Paris traceroute. 1e-05 1e-04 0.001 0.01 0.1 1 0 2 4 6 8 10 12 14 16 18 20 22 Frequency Size of the d-diamonds 1e-05 1e-04 0.001 0.01 0.1 1 0 2 4 6 8 10 12 14 16 18 Frequency Size of the d-diamonds Fig. 15. Distribution of the size of the d -diamond s seen with classic traceroute (left) and P aris tr aceroute (right). This comparison give s an indication of t he impact of p er-ﬂow load balancing on our o bserv ations. Diamonds that disapp ear when switch ing from classic traceroute to Paris traceroute are most probably induced by false links due to p er-ﬂow load balancing. 52 . 6 % of the global diamonds observ ed with t racer- oute disapp ear with P aris traceroute in our data set. This sho ws that diamonds are quite often (approxim ately half of the time) measuremen t artifa cts induced b y p er-ﬂo w load balancing. Studying diamonds also mak es it p ossible to observ e p er-destination load balancing, b y studying the diﬀerences b et w een global diamonds and o ne- destination diamonds. T o understand this, w e mus t ﬁrst notice t ha t all dia- monds indicate a priori the existenc e of alternativ e paths b et w een our source and some rout er b eyond the diamond. This can b e seen a s follows: if a giv en diamond is not a measuremen t artifa ct and do es reﬂect the top ology of the net w ork, then it represen ts alternativ e paths b et w een its head and its tail. F alse diamonds, caused by load bala ncing, also indicate the existence of al- ternativ e paths, but not necessarily bet we en their head and ta il. Consider the example of Fig . 8. In t his case, the existence of three alternative paths b etw een L and G induc es the app eara nce of diamonds. Though there is only one path from L to D , the dia mo nd ( L 0 , D 0 ) indicates the existence of alternativ e paths b et w een some p oin t, a t its head or b efore it (in this case L ), and some ot her p oin t, at its tail or after it (in this case G ). Therefore, global diamonds t hat are not one-destination diamonds, and simi- larly global diamonds that are larger than corresp onding one-destination dia- monds, indicate the existence of a lternativ e paths tha t can b e explored o nly b y probing to w ards diﬀeren t destinations; this corresp onds to p er-destination 23 load balancing. In our case, the diﬀerences b etw een g lobal diamonds and one- destination diamonds are similar for traceroute and P aris tra ceroute: in b oth cases, global diamonds are la rger than one-destination diamonds (this size diﬀerence is muc h more pronounced for traceroute than for P aris traceroute), and g lo bal diamonds that are not one-destination diamonds app ear: there ar e 3,905 such glo ba l diamonds for traceroute, and 3,038 for P aris traceroute. W e can therefore o bserv e a signiﬁcant quan tit y of p er- destination load balancing in our data set. Finally , notice that p er-destination load bala ncing do es not in itself induce false links, and therefore do es not induce false diamonds: these are caused b y the fact that prob es f o r a same measured route follow diﬀerent paths. A load balancer t ha t directs pac k ets dep ending on their destination, how ev er, will direct all prob es of a giv en measured route on the same pa th, b ecause they all hav e the same destination. Our observ ations show ed diﬀeren t things. First, comparing diamo nds obta ined with the classic traceroute and P aris traceroute sho ws that man y diamonds seen with traceroute are measuremen t artifacts caused b y p er-ﬂow load bala nc- ing, and disapp ear when using P aris traceroute. This again show s that p er-ﬂow load balancing ma y hav e a strong impact on one’s observ ations with tracer- oute. Second, comparing global diamo nds with one-destination diamonds al- lo w ed us to ev aluate the amount of p er-destination lo ad bala ncing in our observ at io ns. Although this latter do es not cause measuremen t artif a cts in measured routes, b eing able to detect it leads to inte resting observ ations. Our studies op en the wa y to more detailed analysis of b oth types o f load balancing. W e not e how ev er that, for this type of studies, other meas uremen t strategies will proba bly need to b e emplo y ed. Indeed, as we already men tioned, w e observ ed in our data set t hat a small num ber of dia monds app ear with P aris traceroute but not with classic tra ceroute. As explained in Sec. 4.1, this comes from the fact that some diamonds are observ ed very seldom, a nd migh t therefore b e observ ed o nly with o ne to ol and no t the ot her. Some of these diamonds therefore could po t entially b e o bserv ed with P aris traceroute. A detailed study of loa d balancing should a ddress this question. Concerning diamonds se en with P aris traceroute, w e observ e tha t P aris trace- route is still sub ject to p er-pac k et load balancing. These diamonds may there- fore either reﬂect the real top o logy , or b e false diamonds caused b y p er-pack et load balancing or routing changes during the traceroute. D esigning to ols and measuremen t strategies allo wing the statistical disco v ery of measuremen t a r - tifacts caused b y p er-pack et loa d balancing is an in teresting and c hallenging direction of work; see Sec. 8. 24 6 Other causes of artifacts In the previous section we hav e seen that most lo ops, most cycles, and approx- imately half of the dia mo nds observ ed in traceroute measuremen ts disapp ear when using P aris traceroute. W e will now study t he ones that p ersist. W e will see that P aris traceroute pro vides information that a llo ws us to understand the causes of some of the remaining structures, and in some cases sho ws tha t they also are measuremen t art if acts. The statistics w e pro vide in this section are in regards to the set of structures y et unexplained b y load balancing: unless stated otherwise, t he p ercentages and ratios are to b e t a k en as p ercen tages and ratios among these r emaining structur es . 6.1 Zer o-TTL forw a r ding One explanation for lo ops comes from the traceroute man ual, that men tions a bug in the standard router softw a re of some BSD v ersions: these routers decremen t the TTL of pack ets when it is equal to one, and forw ard the pack et with TTL zer o (whereas a normal r outer would drop the pack et and gener- ate a n ICMP Ti m e Exc e e de d message). Fig . 1 6 shows the consequence of the presence of suc h a router on a route trace. TTL = 9 TTL = 7 TTL = 8 TTL = 6 What we see: Hop 6 Hop 7 Hop 8 Hop 9 0 1 1 0 1 0 1 0 0 0 0 S L B F A L A B Fig. 16. Lo op caused b y a misconﬁ gu r ed router (F) that forw ards pac k ets ha ving TTL 0. W e ma y v alidate the hy p o t hesis o f a misconﬁgured router with a simple ex- p erimen t. Recall that Paris traceroute giv es us the prob e TTL: for a n y lo op caused by zero-TTL forw arding, the prob e TTL of the ﬁrst response from the router in v olv ed in the lo op w ould b e zero. Fig . 17 s hows how P aris traceroute w ould ﬂag the lo op in Fig . 16 with a “!T0”, indicating that the prob e TTL returned for hop 7 w as zero. 6 10.1 0.146.134 452.135 ms 7 10.1 0.127.197 452.246 ms !T0 8 10.1 0.127.197 450.898 ms 9 10.1 0.127.37 451.499 ms Fig. 17. P aris traceroute outpu t for the exa mp le in Fig. 16. 25 W e ran this v a lidation test on all lo ops detected in o ur exp erimen ts. Of the p ersisten t lo op signatures w e observ ed, 68 . 6 % w ere caused b y zero-TTL for- w arding. These represen t 17 . 1% of all the lo op signatures that remained with P aris traceroute. Since p ersisten t lo ops represen t a signiﬁcan t part (63 . 4%) of t he lo o p instances observ ed with P aris traceroute, the num b er of lo op in- stances that remain y et unexplained w as reduced b y 51 . 1 %. 6.2 R outing Cycles F or cycles, the ﬁrst explanation that comes to mind is that the pack ets do follo w a cyclic route (often called a forwar ding lo op , but for clarit y we will av oid this term here), a nd th us that the cycle is not an a rtifact of the measuremen t. The often long span of p erio dic cycles, their p erio dic b eha vior, and the fact that the observ ation of suc h cycles is t ypically asso ciated with a measured route that do es not reac h the destination, argue for this. W e h yp othesize that IP pac k ets that do follo w a cyclic route mo st lik ely do so b ecause of a transien t instabilit y during ro uting con v ergence. 8 One observ a- tion that tends to v alidate this h yp othesis is that almost all p erio dic cycles are transien t, i.e., pack ets follo w a n ormal route for some perio d b efore and after w e observ e the cy cle. Note that this transien t p erio d ma y last longer than for just one route measuremen t: some p erio dic cycles w ere observ ed during a time span of as long as a w eek. Moreo v er, w e used Spring et al.’s [2 9] tec hnique to v erify tha t the iden tical IP addresses whic h fo rm p erio dic cycles really come from the same router. This metho d is based on the fact that a large prop ortion of routers (52% in our data set) use an in ternal coun ter to assign ID ﬁelds to the pac k ets they create, and incremen t it by 1 eve ry time a pac k et is sen t (a mong the 48% of remaining routers in our data set, more than 99% of t hem use a constan t ID v alue set to 0). P ac ke ts sen t within a short time p erio d b y a router using an in ternal coun ter hav e ID v alues that are close in r egards to the 2 16 = 65,536 p ossible v alues of this ﬁeld. Since routers generate far few er pac k ets than they f o rw ard, this me tho d can b e applied on time window s as large as the duratio n of a full measured route (t ypically less than 30 s). While the observ ation o f t w o pack ets ha ving close ID ﬁelds is not a pro of, the accum ulation of suc h observ ations o v er time b ecomes strong evidence. W e applied this metho d – when it was applicable – to o ur p erio dic cycles, since P aris t r aceroute provides the ID ﬁeld of the resp onse pac ke t. W e found a 100 % p ositiv e resp onse, whic h sho ws that the p erio dic cycles w e observ e do in fact corresp ond to the actual routing. 8 After a top olog y c hange, routers ma y ha v e inconsisten t views of top ology du ring some time , whic h is ca lled routing con v ergence. 26 Routing cycles therefore r epresen t 6 0 . 2% of the cycle signatures, and 91 . 6% of the cycle instances. Finally , 43% of the measured r outes with p eriodic cycles actually reac hed the destination. According to a previous study of pack et-lev el measuremen ts in a large ISP back b o ne [30], most routing cycle s last for less than 10 seconds. This time is less than the a v erage time of 30 seconds needed t o measure a route, so it seems reasonable to assume that pac k ets ma y b e caugh t in a cycle for some time, and then reac h the destination. Ho w ev er, the study also found examples of routing cycles that p ersiste d for 10–60 seconds, whic h may b e the explanation for cycles that do not end during our measuremen ts. 6.3 Interrupte d r outes Another explanation f o r lo ops, whic h also applies to cycles, comes from ICMP ‘unreac hable’ resp onses. Some routers, when realizing tha t a pac ke t cannot b e deliv ered to it s destination for some reason, may issue a sp ecial ICMP r e- sp onse, suc h as an ICMP Host Unr e achable , Network Unr e a c hable , or Sour c e Quench (among others), and discard the prob e. When receiving suc h re- sp onses, b oth classic traceroute and P aris tra ceroute consider that the route measuremen t has ende d. Now , these routers may send these sp ecial resp onses ev en after ha ving forw arded one or sev eral prob es. In this case, the ro uter will app ear twice o n the measured route: once when it sends the normal ICMP Time Exc e e de d message, a nd once later when it sends the ICMP ‘unreac hable’ pac k et. W e observ e empirically that, most of the time in these cases, the router sends the unreac hability message just a fter the Time Exc e e de d , leading to the o b- serv atio n of a lo op. But sometimes it ma y also forward sev eral prob e pac k ets b et w een these tw o ev en ts, th us creating a cycle. T r a c king these eve nts is easy , as P aris t r aceroute displa ys the ty p e of ICMP receiv ed as answ ers, whic h allows us to ﬁlt er out these messages. Interrupted routes represen t 62 . 0% of the lo op signatures, and 27 . 0% of the cycle signa- tures. In t erms of instances, these n umbers b ecome resp ectiv ely 19 . 7% and 3 . 5%. The p ortion of instances is comparatively low since this t yp e of mea- suremen t artifact is by essence o ccasional: most routers do not issue this type of me ssage v ery often, and o nly a couple of lo o ps and cycles that w ere caused b y interrupted routes w ere observ ed p ersisten tly . 27 6.4 F ake IPs in r eturn p ackets Our last h yp othesis f o r the presence of lo ops comes from the observ ation of the p ersiste nt n - lo ops. Some subnetw orks are kno wn to b e imp ervious to traceroute measuremen ts b ecause they are b ehind NA T b ox es a nd ﬁrew alls. In these net w orks the routers r eplac e the Source Address ﬁeld of the T i m e Exc e e de d pac k ets on the w a y back to the probing machine , making them app ear to come fro m their b order ga t ewa y . T o v alidate this h yp othesis, w e tried to get some evidence that the consecutiv e replies from seemingly iden tical IP addresses actually came fro m diﬀeren t routers. The simplest w ay is to compare the TTL of the ICMP pack ets they send back: if the TTLs are diﬀeren t, then the resp onses lik ely came fro m diﬀeren t routers. This is even more obvious if these TTLs are alw a ys decreasing with distance (whic h indicates that the answ ering routers are in f a ct aligned on a route.) W e call fak e lo ops the lo ops that a re caused b y this kind of Source Address replacemen t. W e o bserv ed that 6 . 8% of the lo ops signatures a nd 18 . 6% o f the lo op instances came from suc h substituted IPs. The fake lo o ps we detected w ere mostly p ersisten t or systematic. One in teresting result is that most o f the p ersisten t loops detected in our exp erimen t that remained unexplained (i.e., not caused b y zero-TTL forw arding) w ere determined to b e fak e lo ops. Another intere sting result is that all p ersisten t n -lo ops in our data set w ere fak e. The case o f cycles caused b y fak e IPs is mo r e puzzling, since the explanation based on pro tected subnet w orks seems less plausible when observing non- consecutiv e resp onses with the same IP address. There are far few er: only 1 . 6% of the cycle signat ures and 3 . 0% of their instances seemed t o b e caused b y it . 6.5 Summary By considering p er-ﬂow load balancing, zero-TTL forw arding, routing cycles, in terrupted routes, o r fake IPs in return pac k et as causes for artifacts, we w ere able to explain roughly half of the diamonds, and more than 95% of lo ops a nd cycles in our dat a set. T a ble 1 presen ts t he details of the prop ortions of lo o ps (signatures and in- stances), cycles (signatures and instances) and diamonds (global diamonds and one-destination diamonds) caused in our data set by eac h phenomenon. In con trast to the o t her n um b ers presen ted in this s ection, t he p ercentages in 28 Diamonds Loops Cycles global 1-dst sign. inst. sign. inst. Per-ﬂo w load bal. 52.6 56.9 87.1 79.7 68.5 38.4 Zero-TTL fwd. - - 2.21 10.4 - - Routing cycles - - - - 19.0 56.4 Interrup. routes - - 8.00 4.00 8.51 2.16 F ak e IP addresses - - 0.88 3.78 0.50 1.85 Unknown 47.4 43.1 1.81 2.15 3.50 1.17 T otal 100.00 100.00 10 0. 00 100.00 100.00 100.00 T able 1 Summary of ident iﬁed measurement artifacts, sh o wing the prop ortion of observed structures in our data that are attributed to eac h cause. All v alues are p ercen tages. this t able are with respect to al l the structures o bserv ed in our data set: we do no t restrict our selv es to the structures that remain with Paris traceroute. Since not all phenomena are p o ssible causes of app earance for all structures, a dash ‘-’ indicates that a phenomenon do es not apply to a structure. Though our data are not represen tativ e, this illustrates how P aris traceroute helps understand and detect tra ceroute me asuremen t artifa cts. R egarding the unexplained structure s, w e sus p ect, after some preliminary studie s, that mos t are artifacts caused by p er-pack et load balancing. 7 Related w ork W e ha v e earlier published a short v ersion of this pap er [10]. The presen t pap er extends this preliminary work in sev eral w ays : the data set used here is m uc h larger; w e presen t here m uc h more detailed analysis of lo ops and cycles; a nd the discussion on diamonds (Sec. 4.3 and 5.3) is almost completely new. The principal v ar ia n ts on Ja cobson’s tra ceroute [1 ] are G avron’s NANOG traceroute [1 1 ], Eddy’s prtraceroute [31], and T oren’s t cptraceroute [13]. NANOG traceroute a nd prtraceroute b oth lab el IP addresses with the n um b ers o f the ASes to whic h they b elong. Tcptraceroute sends TCP prob es (rather than the classic UDP or IC MP Ec ho prob es) using D estination Port 80 to em ulate w eb traﬃc and th us more easily tra ve rse ﬁrew alls. As described in Sec. 2.3, this has the eﬀect of main taining a constan t ﬂow iden tiﬁer. No prior w ork how ev er has lo ok ed a t the use of this feature to av oid traceroute measuremen t artifacts. Although there is an extensiv e literature on internet maps, and m uc h w ork that uses tra ceroute, there hav e been few studie s o f artifacts as seen from the p ersp ectiv e of traceroute. Mo ors [9] suggests that encoding traceroute prob e pac k et iden tiﬁers in the Source P ort ﬁeld would allo w the use of destination p orts asso ciated with classical services (e.g., HTTP or SMTP). This w ould cause a ll net w ork elemen ts, including load balancers, to handle these pac k ets 29 iden tically to the pack ets of normal net w ork traﬃc. In do ing so, he correctly iden tiﬁes the problem that traceroute may not rep ort routes t ha t no r mal pac k- ets usually follo w. Ho we ve r, the prop osed solution still alters the ﬁv e-tuple, and a to ol built in this w a y would still suﬀer f r o m load balancing and hence w ould presen t the same measuremen t art if acts as classic tra ceroute. P axson’s w ork on end-to-end routing b ehavior in the in ternet [32] uses trace- route to study routing dynamics, including “routing pat ho logies”. Although some of these pathologies do relate to the structures w e discuss in this pa- p er (f or instance, “ro uting lo ops” are one cause of “cycle s”, and “ﬂuttering” is one cause of “diamo nds”), his w ork fo cuses on the routing aspect of his observ at io ns and not on traceroute’s deﬁciencies . T eixeira et al. [33] examine inaccuracies in tro duced in to ISP maps obtained by Ro ck etfuel [4]. Their pa- p er quan tiﬁes diﬀerences b et w een the true and the measured top o logies, and iden tiﬁes ro ut ing changes in the midst of individual traceroutes as b eing re- sp onsible for a p ortio n of the false links in the measured top o logies, but do es not touc h on load balancing. Some top ology inference systems based on tra ceroute a c kno wledge the prob- lem tha t tr a ceroute may rep ort sev eral interfaces at a same ho p on a giv en path and th us ma y infer false links. They handle these problems in diﬀeren t w a ys. Huﬀak er et a l. recognize the problem for skitter [3], but they do not re- p ort a solution. In practice, skitter se nds three prob es p er hop and the routes rep orted b y the arts++ to ol for reading skitter data consist of the ﬁrst address obtained for eac h hop. With R o cketfuel [4 ], Spring et al. attribute a lo w er con- ﬁdence lev el to links inferred from hops that resp ond with m ultiple addres ses. Still, they include all these links in the database from whic h they construct a netw ork’s top olo gy . Their hop e is that a subsequen t alias resolution ste p will elim inate at least some of the false link s. How ever, this only works if load balancing take s place ov er m ultiple links b et we en the same pa ir of routers. 8 Conclusion In this pap er, w e iden tiﬁed and ch ara cterized three structures app earing fre- quen tly in traceroute measuremen ts: lo ops, cycles, and diamonds. W e ex- plained how these structures, some of whic h a re often attributed to routing dynamics o r patholo gies, may rather b e measuremen t artifacts, induced no - tably b y load balancing. W e designe d the P aris traceroute to ol, a new tracer- oute that ﬁnds accurate routes under p er-ﬂo w load balancing and prop osed rigorous metho ds for c hec king the cause of eac h structure. By conducting side- b y-side exp erimen ts with classic traceroute and P aris traceroute, w e w ere able to show that most of these structures a pp earing in our traceroute tra ces are in fa ct measuremen t artifacts, a nd are a v oided with P aris traceroute. Though 30 our measuremen ts are made from one source only and cannot b e considered as statistically represen tativ e o f what one can exp ect in the in ternet in gen- eral, this sho ws that p er-ﬂo w lo a d balancing can hav e a strong impact on one’s observ atio ns, and that the view obtained with P aris traceroute is more accurate. This w ork ma y inspire future researc h in man y directions. First, our exp er- imen ts exp osed other, rarer, structures than those describ ed here, that also deserv e atten tion: w e observ ed cliq ues (sets of in terfaces all link ed to eac h other), dense regions, and IP addresses with a v ery high num b er of links. These structures ar e surprising, and may also b e measuremen t artifacts. Second, w e b eliev e tha t P aris traceroute can b e further impro v ed. W e are w orking on an algorithm to automatically dis cov er all paths betw een a source and a destination in the presence of p er-ﬂow load balancing. W e are a lso dev eloping tec hniques to unco v er accurate routes in the presence of p er-pac k et load balancing. Finally , an impor t an t and in teresting extension of this w ork is to p erform measuremen ts, b oth with classic traceroute and Paris traceroute, from sev eral sources. This w ould make it p ossible to quantify the amount of tra ceroute measuremen t artifacts in the in ternet in general. W e b eliev e that new artifacts and features of interes t will b e observ able when measure ments originate from m ultiple sources. Moreo v er, one of the main conclusions of this w ork b eing that Paris traceroute provid es a m uc h truer view of the top ology than existing to ols, conducting measuremen ts with P aris traceroute will yield data of hig her qualit y for studying the top o logy of the in ternet. Studying the impact of using P aris traceroute on t he observ ed top ological prop erties is a v ery pro mising direction. Ac knowle dgmen ts The P aris tra ceroute to ol w as dev elop ed with ﬁnancial suppo r t from the F renc h CNRS, as part of its contribution to the Europ ean Commission-sp onsored OneL ab pro ject. The researc h using the to ol was conducted with ﬁnancial sup- p ort from the F renc h MNR T AC I S ´ ecurit´ e et Info rmatique 2004 gran t, through the METROSEC pro ject, and fro m the F renc h ANR Jeunes cher cheuses et jeunes cher cheurs 20 05 grant, through the A GRI pro ject. W e thank Eh ud Gav ron a nd Da v e Andersen for their tho ug htful commen ts. W e a lso thank Neil Spring a nd Ratul Maha jan fo r their explanations of how Ro ck e tfuel handles ho ps that resp ond with m ultiple in terfaces. 31 References [1] V. Jacobson, traceroute, the most recen t v ersion is a v ailable at: ftp://ft p.ee.lbl .gov/trac eroute.tar.gz (F ebruary 1989 ). [2] R. Go vindan, H. T angm unaru nkit, Heuristics for in ternet map disco very , in: Pro c. IEEE In fo com, 2000. [3] B. Huﬀaker, D. Plummer, D. Mo ore, k claﬀy , T op olog y disco v ery by activ e probing, in: Pr o c. Sym p osium on Applications and the In ternet, 20 02. [4] N. Spring, R. Maha jan, D. W etherall, Measuring IS P top ologies with Ro c k etfuel, in : Pro c. A CM SIGCOMM, 2002. [5] B. Cheswic k, H. Burc h, In ternet Mapping Pr o ject, h ttp://cm.b ell-labs.com/who/c hes/map/index.h tml (20 00). [6] Y. Sha vitt, E. Shir, DIMES: Let the in ternet measure itself, A CM SI GCOMM Computer Comm unication Revie w 35 (5) (2005) 71 – 74. [7] M. F aloutsos, P . F aloutsos, C. F aloutsos, On p o w er-la w relat ionsh ips of the in ternet top ology , in: Pro c. A CM SIGCOMM, 1999. [8] D. Magoni, M. Ho erdt, Internet core top olog y mapping and analysis, A CM SIGCOMM Computer Communicatio n s 28 (5) (2005) 494 –506. [9] T. Mo ors, Streamlining traceroute by estimating p ath lengths, in: Pro c. IEEE W ork s hop on IP Op erations a n d Managemen t, 2004. [10] B. Augustin, X. Cuv ellier, B. Orgogozo, F. Viger, T. F riedman, M. Latapy , C. Magnien, R. T eixeira, Av oiding t raceroute anomal ies with p aris traceroute., in: Pro c. A CM SIGCOMM In ternet Mea su remen t Conference, 20 06. [11] E. Ga vron , NANOG traceroute, the most rec ent v ersion is a v ailable at: ftp://ft p.login. com/pub/s oftware/traceroute/ (Ma y 1995). [12] W. R. S tev ens, TCP/IP Illustr ated, V olume 1: The P roto cols, Addison-W esley , 1994, Ch. 8, T raceroute Program. [13] M. T oren, tcptraceroute, see http://m ichael.t oren.net/ code/tcptraceroute/ (April 200 1). [14] J. P ostel, In ternet proto col, IETF RF C 791 (Septem b er 19 81). [15] F. Bak er, Requir ements for IP Vers ion 4 routers, IET F RFC 1812 (June 1995). [16] Z. M. Mao, J. Rexford , J. W ang, R. Katz, T o wards an accurate as-lev el traceroute tool, in: P r o c. A CM SIGCO MM, 2003. [17] J. Poste l, Internet control message proto col, IETF RF C 791 (Septem b er 1981). [18] J. Mo y , OSPF V ersion 2, IE TF RFC 2328 (April 1998). [19] R. C allon, Use of OSI IS–IS for Routing in TC P /IP and Dual Environmen ts, IETF RF C 1195 (Decem b er 1990). [20] B. Quoitin, S. Uhlig, C . Pelsser, L. S w innen, O. Bona ve nture, Inte rd omain traﬃc engineering with BGP, IEE E Communicatio n Magazine 41 (5) (2003 ) 122–1 28. [21] Cisco, Ho w do es lo ad balancing w ork?, see http://w ww.cisco .com/en/U S/tech/tk365/technologies_tech_note09186a0080094820.shtml . [22] Ju nip er, Conﬁguring load-balance p er-pac k et action, from the JUNOS 7.0 P olicy F ramew ork Conﬁguration Guideline, see http://w ww.junip er.net/te chpubs/software/junos/junos70/swconfig70- policy/html/policy- ac t i o n s - c o n f i g 1 1 . h t m l . [23] Cisco, Cisco 7600 Series Routers C omm an d R eferen ces, from the Cisco 32 Do cumen tation. [24] R. Go vindan, V. P axson, Estimating r outer IC MP generation dela ys, in: Pro c. of P assiv e and Active Measurement W orkshop, 2002 . [25] S . Bello vin, A tec hnique for counting NA Tted h osts, in: Pr o c. ACM SIGCOMM In ternet Measuremen t W orksh op, 20 02. [26] J. Xia, L. Gao, T. F ei, Flo o din g attac ks b y exploiting p ersistent forwarding lo ops, in: Pro c. A CM SIGCOMM Internet Measurement Con f erence, 2005. [27] Z. M. Mao, D. John son, J. Rexford, J . W ang, R. H. Katz, Scalable and accurate iden tiﬁcation of AS-lev el forw arding p aths, in: Pro c. IEEE Inf o com, 2004. [28] IANA, Sp ecial-use IPv4 addresses, IETF RFC 3330 (Septem b er 2002 ). [29] N. S pring, M. Don tc hev a, M. Ro drig, D. W etherall, Ho w to r esolv e IP aliases, UW CSE T ec hn ical Rep ort (04- 05-04). [30] U. Henga rtn er, S. B. Mo on, R. Mortier, C. Diot, Detection and analysis of routing lo ops in p ack et traces, in: P r o c. ACM SIGCOMM Internet Measurement W ork s hop, 200 2. [31] R. Edd y , prtraceroute, the most rece nt v ersion is part of the In ternet Systems Consortium’s IRR T o olSet, a v ailable fr om: http: //www.is c.org/ (August 1994) . [32] V. P axson, End-to-end internet pac k et dynamics, IEEE/ACM T rans. Net w orking 7 (3) (1999) 277–29 2. [33] R. T eixeira, K. Marzullo, S. S a v age , G. M. V o elk er, In searc h of path div ersity in ISP net w orks, in : Pr o c. A CM SIGCOMM In ternet Measurement Confer en ce, 2003. 33

Detection, Understanding, and Prevention of Traceroute Measurement Artifacts

Original Paper

Comments & Academic Discussion

Leave a Comment

Original Paper

Related Papers

Comments & Academic Discussion

Leave a Comment