Active Ranking using Pairwise Comparisons

Activ e Ranking using Pairwise Comparisons Ke vin G. Jamieson Univ ersity of W isconsin Madison, WI 53706, USA kgjamieson@wisc.edu Robert D. No wak Univ ersity of W isconsin Madison, WI 53706, USA nowak@engr.wisc.edu Abstract This paper examines the problem of ranking a collection of objects using pairwise comparisons (rankings of two objects). In general, the ranking of n objects can be identified by standard sorting methods using n log 2 n pairwise comparisons. W e are interested in natural situations in which relationships among the objects may allow for ranking using far fewer pairwise comparisons. Specifically , we assume that the objects can be embedded into a d -dimensional Euclidean space and that the rankings reflect their relative distances from a common reference point in R d . W e show that under this assumption the number of possible rankings grows like n 2 d and demonstrate an algorithm that can identify a randomly selected ranking using just slightly more than d log n adaptiv ely selected pairwise comparisons, on average. If instead the comparisons are chosen at random, then almost all pairwise comparisons must be made in order to identify any ranking. In addition, we propose a robust, error-tolerant algorithm that only requires that the pairwise comparisons are probably correct. Experimental studies with synthetic and real datasets support the conclusions of our theoretical analysis. 1 Introduction This paper addresses the problem of ranking a set of objects based on a limited number of pair- wise comparisons (rankings between pairs of the objects). A ranking over a set of n objects Θ = ( θ 1 , θ 2 , . . . , θ n ) is a mapping σ : { 1 , . . . , n } → { 1 , . . . , n } that prescribes an order σ (Θ) := θ σ (1) ≺ θ σ (2) ≺ · · · ≺ θ σ ( n − 1) ≺ θ σ ( n ) (1) where θ i ≺ θ j means θ i precedes θ j in the ranking. A ranking uniquely determines the collection of pairwise comparisons between all pairs of objects. The primary objective here is to bound the number of pairwise comparisons needed to correctly determine the ranking when the objects (and hence rankings) satisfy certain known structural constraints. Specifically , we suppose that the objects may be embedded into a low-dimensional Euclidean space such that the ranking is consistent with distances in the space. W e wish to exploit such structure in order to discov er the ranking using a very small number of pairwise comparisons. T o the best of our kno wledge, this is a previously open and unsolved problem. There are practical and theoretical motiv ations for restricting our attention to pairwise rankings that are discussed in Section 2. W e begin by assuming that ev ery pairwise comparison is consistent with an unknown ranking. Each pairwise comparison can be viewed as a query: is θ i before θ j ? Each query provides 1 bit of information about the underlying ranking. Since the number of rankings is n ! , in general, specifying a ranking requires Θ( n log n ) bits of information. This implies that at least this many pairwise comparisons are required without additional assumptions about the ranking. In fact, this lo wer bound can be achiev ed with a standard adaptive sorting algorithm like binary sort [1]. In large-scale problems or when humans are queried for pairwise comparisons, obtaining this many pairwise comparisons may be impractical and therefore we consider situations in which the space of rankings is structured and thereby less complex. 1 A natural way to induce a structure on the space of rankings is to suppose that the objects can be embedded into a d -dimensional Euclidean space so that the distances between objects are consistent with the ranking. This may be a reasonable assumption in many applications, and for instance the audio dataset used in our experiments is believed to have a 2 or 3 dimensional embedding [2]. W e further discuss moti v ations for this assumption in Section 2. It is not dif ficult to sho w (see Section 3) that the number of full rankings that could arise from n objects embedded in R d grows like n 2 d , and so specifying a ranking from this class requires only O ( d log n ) bits. The main results of the paper show that under this assumption a randomly selected ranking can be determined using O ( d log n ) pairwise comparisons selected in an adapti ve and sequential fashion, but almost all  n 2  pairwise rankings are needed if they are picked randomly rather than selectively . In other words, actively selecting the most informative queries has a tr emendous impact on the complexity of learning the corr ect ranking . 1.1 Problem statement Let σ denote the ranking to be learned. The objectiv e is to learn the ranking by querying the reference for pairwise comparisons of the form q i,j := { θ i ≺ θ j } . (2) The response or label of q i,j is binary and denoted as y i,j := 1 { q i,j } where 1 is the indicator function; ties are not allowed. The main results quantify the minimum number of queries or labels required to determine the reference’ s ranking, and they are based on two key assumptions. A1 Embedding: The set of n objects are embedded in R d (in general position) and we will also use θ 1 , . . . , θ n to refer to their (kno wn) locations in R d . Every ranking σ can be specified by a r efer ence point r σ ∈ R d , as follo ws. The Euclidean distances between the reference and objects are consistent with the ranking in the following sense: if the σ ranks θ i ≺ θ j , then k θ i − r σ k < k θ j − r σ k . Let Σ n,d denote the set of all possible rankings of the n objects that satisfy this embedding condition. The interpretation of this assumption is that we kno w how the objects are related (in the embedding), which limits the space of possible rankings. The ranking to be learned, specified by the reference (e.g., preferences of a human subject), is unknown. Many have studied the problem of finding an embedding of objects from data [3, 4, 5]. This is not the focus here, but it could certainly play a supporting role in our methodology (e.g., the embedding could be determined from known similarities between the n objects, as is done in our experiments with the audio dataset). W e assume the embedding is giv en and our interest is minimizing the number of queries needed to learn the ranking, and for this we require a second assumption. A2 Consistency: Every pairwise comparison is consistent with the ranking to be learned. That is, if the reference ranks θ i ≺ θ j , then θ i must precede θ j in the (full) ranking. As we will discuss later in Section 3.2, these two assumptions alone are not enough to rule out pathological arrangements of objects in the embedding for which at least Ω( n ) queries must be made to recov er the ranking. Howe v er , because such situations are not representativ e of what is typically encountered, we analyze the problem in the framew ork of the av erage-case analysis [6]. Definition 1. W ith each ranking σ ∈ Σ n,d we associate a pr obability π σ such that P σ ∈ Σ n,d π σ = 1 . Let π denote these pr obabilities and write σ ∼ π for shorthand. The uniform distribution corr esponds to π σ = | Σ n,d | − 1 for all σ ∈ Σ n,d , and we write σ ∼ U for this special case. Definition 2. If M n ( σ ) denotes the number of pairwise comparisons r equested by an algorithm to identify the ranking σ , then the average query comple xity with r espect to π is denoted by E π [ M n ] . The main results are prov en for the special case of π = U , the uniform distribution, to make the analysis more transparent and intuitiv e. Howe ver the results can easily be extended to general dis- tributions π that satisfy certain mild conditions (see Appendix A.5). All results henceforth, unless otherwise noted, will be giv en in terms of (uniform) a verage query complexity and we will say such results hold “on av erage. ” Our main results can be summarized as follows. If the queries are chosen deterministically or ran- domly in advance of collecting the corresponding pairwise comparisons, then we show that almost all  n 2  pairwise comparisons queries are needed to identify a ranking under the assumptions above. Howe v er , if the queries are selected in an adaptiv e and sequential fashion according to the algorithm 2 Query Selection Algorithm input: n objects in R d initialize: objects Θ = { θ 1 , . . . , θ n } in uni- formly random order for j=2,. . . ,n for i=1,. . . ,j-1 if q i,j is ambiguous , request q i,j ’ s label from reference; else impute q i,j ’ s label from previously labeled queries. output: ranking of n objects Figure 1: Sequential algorithm for selecting queries. See Figure 2 and Section 4.2 for the definition of an ambiguous query . θ 1 θ 2 θ 3 q 1 , 2 q 1 , 3 q 2 , 3 Figure 2: Objects θ 1 , θ 2 , θ 3 and queries. The r σ lies in the shaded region (consistent with the labels of q 1 , 2 , q 1 , 3 , q 2 , 3 ). The dotted (dashed) lines represent ne w queries whose labels are (are not) ambiguous giv en those labels. in Figure 1, then we show that the number of pairwise rankings required to identify a ranking is no more than a constant multiple of d log n , on average. The algorithm requests a query if and only if the corresponding pairwise ranking is ambiguous (see Section 4.2), meaning that it cannot be determined from previously collected pairwise comparisons and the locations of the objects in R d . The ef ficiency of the algorithm is due to the fact that most of the queries are unambiguous when considered in a sequential fashion. For this very same reason, picking queries in a non-adaptiv e or random fashion is very inefficient. It is also noteworth y that the algorithm is also computationally efficient with an overall complexity no greater than O ( n poly ( d ) poly (log n )) (see Appendix A.1). In Section 5 we present a robust version of the algorithm of Figure 1 that is tolerant to a fraction of errors in the pairwise comparison queries. In the case of persistent error s (see Section 5) we show that we can find a probably approximately correct ranking by requesting just O ( d log 2 n ) pairwise comparisons. This allo ws us to handle situations in which either or both of the assumptions, A1 and A2 , are reasonable approximations to the situation at hand, b ut do not hold strictly (which is the case in our experiments with the audio dataset). Proving the main results inv olves an uncommon marriage of ideas from the ranking and statistical learning literatures. Geometrical interpretations of our problem deri ve from the seminal works of [7] in ranking and [8] in learning. From this perspecti ve our problem bears a strong resemblance to the halfspace learning problem, with two crucial distinctions. In the ranking problem, the under- lying halfspaces are not in general position and hav e strong dependencies with each other . These dependencies in validate man y of the typical analyses of such problems [9, 10]. One popular method of analysis in exact learning inv olv es the use of something called the extended teaching dimension [11]. Howe ver , because of the possible pathological situations alluded to earlier , it is easy to show that the extended teaching dimension must be at least Ω( n ) making that sort of worst-case analysis uninteresting. These differences present unique challenges to learning. 2 Motivation and r elated work The problem of learning a ranking from few pairwise comparisons is motiv ated by what we perceiv e as a significant gap in the theory of ranking and permutation learning. Most work in ranking with structural constraints assumes a passiv e approach to learning; pairwise comparisons or partial rank- ings are collected in a random or non-adapti ve fashion and then aggregated to obtain a full ranking (cf. [12, 13, 14, 15]). Howe ver , this may be quite inef ficient in terms of the number of pairwise com- parisons or partial rankings needed to learn the (full) ranking. This inefficiency was recently noted in the related area of social choice theory [16]. Furthermore, empirical evidence suggests that, e ven under complex ranking models, adapti vely selecting pairwise comparisons can reduce the number needed to learn the ranking [17]. It is cause for concern since in many applications it is expen- siv e and time-consuming to obtain pairwise comparisons. For example, psychologists and mark et researchers collect pairwise comparisons to gauge human preferences ov er a set of objects, for sci- entific understanding or product placement. The scope of these experiments is often very limited 3 simply due to the time and expense required to collect the data. This suggests the consideration of more selecti ve and judicious approaches to gathering inputs for ranking. W e are interested in taking advantage of underlying structure in the set of objects in order to choose more informati ve pairwise comparison queries. From a learning perspectiv e, our work adds an active learning component to a problem domain that has primarily been treated from a passiv e learning mindset. W e focus on pairwise comparison queries for two reasons. First, pairwise comparisons admit a halfspace representation in embedding spaces which allows for a geometrical approach to learning in such structured ranking spaces. Second, pairwise comparisons are the most common form of queries in many applications, especially those in v olving human subjects. For example, consider the problem of finding the most highly ranked object, as illustrated by the following familiar task. Suppose a patient needs a new pair of prescription eye lenses. Faced with literally millions of possible prescriptions, the doctor will present candidate prescriptions in a sequential fashion followed by the query: better or worse? Even if certain queries are repeated to account for possible inaccurate answers, the doctor can locate an accurate prescription with just a handful of queries. This is possible presumably because the doctor understands (at least intuitiv ely) the intrinsic space of prescriptions and can efficiently search through it using only binary responses from the patient. W e assume that the objects can be embedded in R d and that the distances between objects and the reference are consistent with the ranking (Assumption A1 ). The problem of learning a general function f : R d → R using just pairwise comparisons that correctly ranks the objects embedded in R d has previously been studied in the passi ve setting [12, 13, 14, 15]. The main contributions of this paper are theoretical bounds for the specific case when f ( x ) = || x − r σ || where r σ ∈ R d is the reference point. This is a standard model used in multidimensional unfolding and psychometrics [7, 18] and one can show that this model also contains the familiar functions f ( x ) = r T σ x for all r σ ∈ R d . W e are unaware of any existing query-complexity bounds for this problem. W e do not assume a generati ve model is responsible for the relationship between rankings to embeddings, but one could. For example, the objects might have an embedding (in a feature space) and the ranking is generated by distances in this space. Or alternatively , structural constraints on the space of rankings could be used to generate a consistent embedding. Assumption A1 , while arguably quite natural/reasonable in many situations, significantly constrains the set of possible rankings. 3 Geometry of rankings from pairwise comparisons The embedding assumption A1 gives rise to geometrical interpretations of the ranking problem, which are dev eloped in this section. The pairwise comparison q i,j can be viewed as the membership query: is θ i ranked before θ j in the (full) ranking σ ? The geometrical interpretation is that q i,j re- quests whether the reference r σ is closer to object θ i or object θ j in R d . Consider the line connecting θ i and θ j in R d . The hyperplane that bisects this line and is orthogonal to it defines two halfspaces: one containing points closer to θ i and the other the points closer to θ j . Thus, q i,j is a membership query about which halfspace r σ is in, and there is an equiv alence between each query , each pair of objects, and the corresponding bisecting hyperplane. The set of all possible pairwise comparison queries can be represented as  n 2  distinct halfspaces in R d . The intersections of these halfspaces partition R d into a number of cells, and each one corresponds to a unique ranking of Θ . Arbitrary rankings are not possible due to the embedding assumption A1 , and recall that the set of rankings possible under A1 is denoted by Σ n,d . The cardinality of Σ n,d is equal to the number of cells in the partition. W e will refer to these cells as d -cells (to indicate they are subsets in d -dimensional space) since at times we will also refer to lower dimensional cells; e.g., ( d − 1) -cells. 3.1 Counting the number of possible rankings The following lemma determines the cardinality of the set of rankings, Σ n,d , under assumption A1 . Lemma 1. [7] Assume A1-2 . Let Q ( n, d ) denote the number of d -cells defined by the hyperplane ar- rangement of pairwise comparisons between these objects (i.e . Q ( n, d ) = | Σ n,d | ). Q ( n, d ) satisfies the r ecursion Q ( n, d ) = Q ( n − 1 , d ) + ( n − 1) Q ( n − 1 , d − 1) , where Q (1 , d ) = 1 and Q ( n, 0) = 1 . (3) 4 In the hyperplane arrangement induced by the n objects in d dimensions, each hyperplane is inter- sected by ev ery other and is partitioned into Q ( n − 1 , d − 1) subsets or ( d − 1) -cells. The recursion, abov e, arises by considering the addition of one object at a time. Using this lemma in a straightfor- ward fashion, we pro ve the follo wing corollary in Appendix A.2. Corollary 1. Assume A1-2 . Ther e exist positive r eal numbers k 1 and k 2 such that k 1 n 2 d 2 d d ! < Q ( n, d ) < k 2 n 2 d 2 d d ! for n > d + 1 . If n ≤ d + 1 then Q ( n, d ) = n ! . F or n suf ficiently lar ge, k 1 = 1 and k 2 = 2 suffice . 3.2 Lower bounds on query complexity Since the cardinality of the set of possible rankings is | Σ n,d | = Q ( n, d ) , we hav e a simple lower bound on the number of queries needed to determine the ranking. Theorem 1. Assume A1-2 . T o reconstruct an arbitr ary ranking σ ∈ Σ n,d any algorithm will r equire at least log 2 | Σ n,d | = Θ(2 d log 2 n ) pairwise comparisons. Pr oof. By Corollary 1 | Σ n,d | = Θ( n 2 d ) , and so at least 2 d log n bits are needed to specify a ranking. Each pairwise comparison provides at most one bit. If each query provides a full bit of information about the ranking, then we achiev e this lower bound. For example, in the one-dimensional case ( d = 1 ) the objects can be ordered and binary search can be used to select pairwise comparison queries, achieving the lower bound. This is generally impossible in higher dimensions. Even in two dimensions there are placements of the objects (still in general position) that produce d -cells in the partition induced by queries that ha ve n − 1 faces (i.e., bounded by n − 1 hyperplanes) as shown in Appendix A.3. It follows that the worst case situation may require at least n − 1 queries in dimensions d ≥ 2 . In light of this, we conclude that worst case bounds may be o verly pessimistic indications of the typical situation, and so we instead consider the av erage case performance introduced in Section 1.1. 3.3 Inefficiency of random queries The geometrical representation of the ranking problem reveals that randomly choosing pairwise comparison queries is inefficient relativ e to the lower bound abov e. T o see this, suppose m queries were chosen uniformly at random from the possible  n 2  . The answers to m queries narro ws the set of possible rankings to a d -cell in R d . This d -cell may consist of one or more of the d -cells in the partition induced by all queries. If it contains more than one of the partition cells, then the underlying ranking is ambiguous. Theorem 2. Assume A1-2 . Let N =  n 2  . Suppose m pairwise comparison ar e chosen uniformly at random without replacement fr om the possible  n 2  . Then for all positive integ ers N ≥ m ≥ d the pr obability that the m queries yield a unique r anking is  m d  /  N d  ≤ ( em N ) d . Pr oof . No fewer than d hyperplanes bound each d -cell in the partition of R d induced by all possible queries. The probability of selecting d specific queries in a random draw of m is equal to  N − d m − d  N m  =  m d  N d  ≤ m d d ! d d N d ≤  m N  d d d d ! ≤  em N  d .  Note that  m d  /  N d  < 1 / 2 unless m = Ω( n 2 ) . Therefore, if the queries are randomly chosen, then we will need to ask almost all queries to guarantee that the inferred ranking is probably correct. 4 Analysis of sequential algorithm f or query selection Now consider the basic sequential process of the algorithm in Figure 1. Suppose we ha ve ranked k − 1 of the n objects. Call these objects 1 through k − 1 . This places the reference r σ within a d -cell (defined by the labels of the comparison queries between objects 1 , . . . , k − 1 ). Call this d -cell C k − 1 . No w suppose we pick another object at random and call it object k . A comparison 5 query between object k and one of objects 1 , . . . , k − 1 can only be informativ e (i.e., ambiguous) if the associated hyperplane intersects this d -cell C k − 1 (see Figure 2). If k is significantly larger than d , then it turns out that the cell C k − 1 is probably quite small and the probability that one of the queries intersects C k − 1 is very small; in fact the probability is on the order of 1 /k 2 . 4.1 Hyperplane-point duality Consider a hyperplane h = ( h 0 , h 1 , . . . , h d ) with ( d + 1) parameters in R d and a point p = ( p 1 , . . . , p d ) ∈ R d that does not lie on the hyperplane. Checking which halfspace p falls in, i.e., h 1 p 1 + h 2 p 2 + · · · + h d p d + h 0 ≷ 0 , has a dual interpretation: h is a point in R d +1 and p is a hyperplane in R d +1 passing through the origin (i.e., with d free parameters). Recall that each possible ranking can be represented by a reference point r σ ∈ R d . Our problem is to determine the ranking, or equiv alently the vector of responses to the  n 2  queries represented by hyperplanes in R d . Using the abov e observation, we see that our problem is equiv alent to finding a labeling over  n 2  points in R d +1 with as few queries as possible. W e will refer to this alternati ve representation as the dual and the former as the primal. 4.2 Characterization of an ambiguous query The characterization of an ambiguous query has interpretations in both the primal and dual spaces. W e will now describe the interpretation in the dual which will be critical to our analysis of the sequential algorithm of Figure 1. Definition 3. [8] Let S be a finite subset of R d and let S + ⊂ S be points labeled +1 and S − = S \ S + be the points labeled − 1 and let x be any other point except the origin. If there exists two homogeneous linear separators of S + and S − that assign differ ent labels to the point x , then the label of x is said to be ambiguous with respect to S. Lemma 2. [8, Lemma 1] The label of x is ambiguous with respect to S if and only if S + and S − ar e homogeneously linearly separ able by a ( d − 1) -dimensional subspace containing x . Let us consider the implications of this lemma to our scenario. Assume that we hav e labels for all the pairwise comparisons of k − 1 objects. Ne xt consider a new object called object k . In the dual, the pairwise comparison between object k and object i , for some i ∈ { 1 , . . . , k − 1 } , is ambiguous if and only if there exists a hyperplane that still separates the original points and also passes through this new point. In the primal, this separating hyperplane corresponds to a point lying on the hyperplane defined by the associated pairwise comparison. 4.3 The probability that a query is ambiguous An essential component of the sequential algorithm of Figure 1 is the initial random order of the objects; every sequence in which it could consider objects is equally probable. This allo ws us to state a nontrivial f act about the partial rankings of the first k objects observed in this sequence. Lemma 3. Assume A1-2 and σ ∼ U . Consider the subset S ⊂ Θ with | S | = k that is randomly selected fr om Θ such that all  n k  subsets are equally pr obable. If Σ k,d denotes the set of possible rankings of these k objects then every σ ∈ Σ k,d is equally pr obable . Pr oof. Let a k -partition denote the partition of R d into Q ( k , d ) d -cells induced by k objects for 1 ≤ k ≤ n . In the n -partition, each d -cell is weighted uniformly and is equal to 1 /Q ( n, d ) . If we uniformly at random select k objects from the possible n and consider the k -partition, each d -cell in the k -partition will contain one or more d -cells of the n -partition. If we select one of these d -cells from the k -partition, on av erage there will be Q ( n, d ) /Q ( k , d ) d -cells from the n -partition contained in this cell. Therefore the probability mass in each d -cell of the k -partition is equal to the number of cells from the n -partition in this cell multiplied by the probability of each of those cells from the n -partition: Q ( n, d ) /Q ( k , d ) × 1 /Q ( n, d ) = 1 /Q ( k , d ) , and | Σ k,d | = Q ( k , d ) . As described above, for 1 ≤ i ≤ k some of the pairwise comparisons q i,k +1 may be ambiguous. The algorithm chooses a random sequence of the n objects in its initialization and does not use the labels of q 1 ,k +1 , . . . , q j − 1 ,k +1 , q j +1 ,k +1 , . . . , q k,k +1 to make a determination of whether or not q j,k +1 is ambiguous. It follows that the ev ents of requesting the label of q i,k +1 for i = 1 , 2 , . . . , k 6 are independent and identically distributed (conditionally on the results of queries from previous steps). Therefore it makes sense to talk about the probability of requesting any one of them. Lemma 4. Assume A1-2 and σ ∼ U . Let A ( k, d, U ) denote the pr obability of the event that the pairwise comparison q i,k +1 is ambiguous for i = 1 , 2 , . . . , k . Then there exists a positive, real number constant a independent of k such that for k ≥ 2 d , A ( k , d, U ) ≤ a 2 d k 2 . Pr oof. By Lemma 2, a point in the dual (pairwise comparison) is ambiguous if and only if there exists a separating hyperplane that passes through this point. This implies that the hyperplane rep- resentation of the pairwise comparison in the primal intersects the cell containing r σ (see Figure 2 for an illustration of this concept). Consider the partition of R d generated by the hyperplanes cor- responding to pairwise comparisons between objects 1 , . . . , k . Let P ( k , d ) denote the number of d -cells in this partition that are intersected by a hyperplane corresponding to one of the queries q i,k +1 , i ∈ { 1 , . . . , k } . Then it is not dif ficult to sho w that P ( k , d ) is bounded above by a con- stant independent of n and k times k 2( d − 1) 2 d − 1 ( d − 1)! (see Appendix A.4). By Lemma 3, e very d -cell in the partition induced by the k objects corresponds to an equally probable ranking of those objects. Therefore, the probability that a query is ambiguous is the number of cells intersected by the corre- sponding hyperplane divided by the total number of d -cells, and therefore A ( k, d, U ) = P ( k,d ) Q ( k,d ) . The result follows immediately from the bounds on P ( k, d ) and Corollary 1. Because the individual ev ents of requesting each query are conditionally independent, the total num- ber of queries requested by the algorithm is just M n = P n − 1 k =1 P k i =1 1 { Request q i,k +1 } . Using the results abov e, it straightforward to prov e the main theorem belo w (see Appendix A.5). Theorem 3. Assume A1-2 and σ ∼ U . Let the random variable M n denote the number of pairwise comparisons that ar e r equested in the algorithm of F igur e 1, then E U [ M n ] ≤ d 2 da e log 2 n. Furthermor e, if σ ∼ π and max σ ∈ Σ n,d π σ ≤ c | Σ n,d | − 1 for some c > 0 , then E π [ M n ] ≤ c E U [ M n ] . 5 Robust sequential algorithm f or query selection W e now extend the algorithm of Figure 1 to situations in which the response to each query is only probably correct. If the correct label of a query q i,j is y i,j , we denote the possibly incorrect response by Y i,j . Let the probability that Y i,j = y i,j be equal to 1 − p , p < 1 / 2 . The robust algorithm operates in the same fashion as the algorithm in Figure 1, with the exception that when an ambiguous query is encountered sev eral (equiv alent) queries are made and a decision is based on the majority vote. W e will now judge performance based on two metrics: (i) how many queries are requested and (ii) how accurate the estimated ranking is with respect to the true ranking before it was corrupted. For any two rankings σ , b σ we adopt the popular Kendell-T au distance [19] d τ ( σ, b σ ) = X ( i,j ): σ ( i ) <σ ( j ) 1 { b σ ( j ) < b σ ( i ) } (4) where 1 is the indicator function. Clearly , d τ ( σ, b σ ) = d τ ( b σ, σ ) and 0 ≤ d τ ( σ, b σ ) ≤  n 2  . For any ranking σ ∈ Σ n,d we wish to find an estimate b σ ∈ Σ n,d that is close in terms of d τ ( σ, b σ ) without requesting too many pairwise comparisons. F or con venience, we will some times report results in terms of the proportion  of incorrect pairwise orderings such that d τ ( σ, b σ ) ≤   n 2  . Using the equiv alence of the K endell-T au and Spearman’ s footrule distances (see [20]), if d τ ( σ, b σ ) ≤   n 2  then each object in b σ is, on av erage, no more than O ( n ) positions away from its position in σ . Thus, the Kendell-T au distance is an intuiti v e measure of closeness between two rankings. First consider the case in which each query can be repeated to obtain multiple independent responses (votes) for each comparison query . This random err ors model arises, for example, in social choice theory where the “reference” is a group of people, each casting a vote. The elementary proof of the next theorem is gi ven in Appendix A.6. 7 Theorem 4. Assume A1-2 and σ ∼ U but that each response to the query q i,j is a r ealization of an i.i.d. Bernoulli random variable Y i,j with P ( Y i,j 6 = y i,j ) ≤ p < 1 / 2 for all distinct i, j ∈ { 1 , . . . , n } . If all ambiguous queries are decided by the majority vote of R independent responses to each such query , then with probability gr eater than 1 − 2 n log 2 ( n ) exp( − 1 2 (1 − 2 p ) 2 R ) this pr ocedur e corr ectly identifies the corr ect ranking (i.e.  = 0 ) and r equests no mor e than O ( Rd log n ) queries on averag e. W e can deduce from the abov e theorem that to exactly recover the true ranking under the stated conditions with probability 1 − δ , one need only request O  d (1 − 2 p ) − 2 log 2 ( n/δ )  pairwise com- parisons, on av erage. In other situations, if we ask the same query multiple times we may get the same, possibly incorrect, response each time. This persistent err ors model is natural, for example, if the reference is a single human. Under this model, if two rankings dif fer by only a single pairwise comparison, then they cannot be distinguished with probability greater than 1 − p . So, in general, exact recov ery of the ranking cannot be guaranteed with high probability . The best we can hope for is to exactly recover a partial ranking of the objects (i.e. the ranking over a subset of the objects) or a ranking that is merely probably approximately correct in terms of the Kendell-T au distance of (4) . W e will first consider the task of exact recov ery of a partial ranking of objects and then turn our attention to the recov ery of an approximate ranking. Henceforth, we will assume the errors are persistent. 5.1 Robust sequential algorithm f or persistent err ors The rob ust query selection algorithm for persistent errors is presented in Figure 3. The ke y ingredient in the persistent errors setting is the design of a voting set for each ambiguous query encountered. Suppose the query q i,j is ambiguous in the algorithm of Figure 1. In principle, a voting set could be constructed using objects ranked between i and j . If object k is between i and j , then note that y i,j = y i,k = y k,j . In practice, we cannot identify the subset of objects ranked between i and j exactly , but we can find a set that contains them. For an ambiguous query q i,j define T i,j := { k ∈ { 1 . . . , n } : q i,k , q k,j , or both are ambiguous } . (5) Then T i,j contains all objects ranked between i and j (if k is ranked between i and j , and q i,k and q k,j are unambiguous, then so is q i,j , a contradiction). Furthermore, if the first j − 1 objects ranked in the algorithm were selected uniformly at random (or initialized in a random order in the algorithm) Lemma 3 implies that each object in T i,j is ranked between i and j with probability at least 1 / 3 due to the uniform distribution ov er the rankings Σ n,d (see Appendix A.7 for an explanation). T i,j will be our voting set. If we follow the sequential procedure of the algorithm of Figure 3, the first query encountered, call it q 1 , 2 , will be ambiguous and T 1 , 2 will contain all the other n − 2 objects. Howe ver , at some point for some query q i,j it will become probable that the objects i and j are closely ranked. In that case, T i,j may be rather small, and so it is not always possible to find a sufficiently large voting set to accurately determine y i,j . Therefore, we must specify a size- threshold R ≥ 0 . If the size of T i,j is at least R , then we draw R indices from T i,j uniformly at random without replacement, call this set { t l } R l =1 , and decide the label for q i,j by voting over the responses to { q i,k , q k,j : k ∈ { t l } R l =1 } ; otherwise we pass ov er object j and mo ve on to the next object in the list. Giv en that | T i,j | ≥ R the label of q i,j is determined by: 0 i ≺ j R j ≺ i X k ∈{ t l } R l =1 1 { Y i,k = 1 ∧ Y k,j = 1 } − 1 { Y i,k = 0 ∧ Y k,j = 0 } . (6) In the next section we will analyze this algorithm and show that it enjoys a very fa vorable query complexity while also admitting a probably approximately correct ranking. 5.2 Analysis of the rob ust sequential algorithm Consider the robust algorithm in Figure 3. At the end of the process, some objects that were passed ov er may then be unambiguously ranked (based on queries made after they were passed over) or they can be ranked without v oting (and without guarantees). As mentioned in Section 5.1, if the first j − 1 objects ranked in the algorithm of Figure 3 were chosen uniformly at random from the full set (i.e., none of the first j − 1 objects were passed over) then there is at least a one in three chance each 8 Robust Query Selection Algorithm input: n objects in R d , R ≥ 0 initialize: objects Θ = { θ 1 , . . . , θ n } in uniformly random order , Θ 0 = Θ for j=2,. . . ,n for i=1,. . . ,j-1 if q i,j is ambiguous , T i,j := { k ∈ { 1 . . . , n } : q i,k , q k,j , or both are ambiguous } if | T i,j | ≥ R { t l } R l =1 i.i.d. ∼ uniform ( T i,j ) . request Y i,k , Y k,j for all k ∈ { t l } R l =1 decide label of q i,j with (6) else Θ 0 ← Θ 0 \ θ j , j ← j + 1 else impute q i,j ’ s label from previously labeled queries. output: ranking over objects in Θ 0 Figure 3: Robust sequential algorithm for selecting queries of Sec- tion 5.1. See Figure 2 and Section 4.2 for the definition of an am- biguous query . object in T i,j for some ambiguous query q i,j is ranked between i and j . W ith this in mind, we have the following theorem, pro ved in Appendix A.7. Theorem 5. Assume A1-2 , σ ∼ U , and P ( Y i,j 6 = y i,j ) = p . F or every set T i,j constructed in the algorithm of F igur e 3, assume that an object selected uniformly at random fr om T i,j is ranked between θ i and θ j with pr obability at least 1 / 3 . Then for any size-thr eshold R ≥ 1 , with pr obability gr eater than 1 − 2 n log 2 ( n ) exp  − 2 9 (1 − 2 p ) 2 R  the algorithm corr ectly ranks at least n/ (2 R + 1) objects and r equests no mor e than O ( R d log n ) queries on aver age . Note that before the algorithm skips ov er an object for the first time, all objects that are ranked at such an intermediate stage are a subset chosen uniformly at random from the full set of objects, due to the initial randomization. Therefore, if T i,j is a v oting set in this stage, an object selected uniformly at random from T i,j is ranked between θ i and θ j with probability at least 1 / 3 , per Lemma 3. After one or more objects are passed ov er , howe ver , the distribution is no longer necessarily uniform due to this action, and so the assumption of the theorem above may not hold. The procedure of the algorithm is still reasonable, but it is dif ficult to giv e guarantees on performance without the assumption. Nev ertheless, this discussion leads us to wonder how many objects the algorithm will rank before it skips ov er its first object. The next lemma is prov ed in Appendix A.8. Lemma 5. Consider a ranking of n objects and suppose objects are drawn sequentially , cho- sen uniformly at random without r eplacement. If M is the largest inte ger suc h that M ob- jects ar e drawn before any object is within R positions of another one in the ranking , then M ≥ q n/R 6 log(2) with pr obability at least 1 6 log(2)  e − ( √ 6 log(2) R/n +1) 2 / 2 − 2 − n/ (3 R )  . As n/R → ∞ , P ( M ≥ q n/R 6 log(2) ) → 1 6 √ e log(2) . Lemma 5 characterizes ho w many objects the rob ust algorithm will rank before it passes over its first object because if there are at least R objects between ev ery pair of the first M objects, then T i,j ≥ R for all distinct i, j ∈ { 1 , . . . , M } and none of the first M objects will be passed ov er . W e can conclude from Lemma 5 and Theorem 5 that with constant probability (with respect to the initial or- dering of the objects and the randomness of the v oting), the algorithm of Figure 3 e xactly reco vers a partial ranking of at least Ω( p (1 − 2 p ) 2 n/ log n ) objects by requesting just O  d (1 − 2 p ) − 2 log 2 n  pairwise comparisons, on av erage, with respect to all the rankings in Σ n,d . If we repeat the algorithm with different initializations of the objects each time, we can boost this constant probability to an arbitrarily high probability (recall that the responses to queries will not change over the repetitions). Note, howe ver , that the correctness of the partial ranking does not indicate ho w approximately cor- rect the remaining rankings will be. If the algorithm of Figure 3 ranks m objects before skipping 9 0 10 20 30 40 50 60 70 80 90 100 0 100 200 300 400 500 600 log 2 | Σ n,d | 2l o g 2 | Σ n,d | Di me ns i on Numb er o f query requ ests Figure 4: Mean and standard de viation of re- quested queries (solid) in the error-free case for n = 100 ; log 2 | Σ n,d | is a lower bound (dashed). T able 1: Statistics for the algorithm robust to persistent errors of Section 5 with respect to all  n 2  pairwise comparisons. Recall y is the noisy response vector , ˜ y is the embedding’ s solution, and ˆ y is the output of the robust al- gorithm. Dimension 2 3 % of queries requested mean 14.5 18.5 std 5.3 6 A verage error d ( y , ˜ y ) 0.23 0.21 d ( y , ˆ y ) 0.31 0.29 ov er its first, then the next lemma quantifies how accurate an estimated ranking is in terms of Kendel- T au distance, giv en that it is some ranking in Σ n,d that is consistent with the probably correct partial ranking of the first m objects (the output ranking of the algorithm may contain more than m objects but we mak e no guarantees about these additional objects). The proof is av ailable in Appendix A.9. Lemma 6. Assume A1-2 and σ ∼ U . Suppose we select 1 ≤ m < n objects uniformly at random fr om the n and corr ectly r ank them amongst themselves. If b σ is any ranking in Σ n,d that is consistent with all the known pairwise comparisons between the m objects, then E [ d τ ( σ, b σ )] = O ( d/m 2 )  n 2  , wher e the expectation is with r espect to the random selection of objects and the distribution of the rankings U . Combining Lemmas 5 and 6 in a straightforward way , we have the follo wing theorem. Theorem 6. Assume A1-2 , σ ∼ U , and P ( Y i,j 6 = y i,j ) = p . If R = Θ((1 − 2 p ) − 2 log n ) and b σ is any ranking in Σ n,d that is consistent with all known pairwise comparisons between the sub- set of objects ranked in the output of the algorithm of F igur e 3, then with constant probability E [ d τ ( σ, b σ )] = O ( d (1 − 2 p ) − 2 log( n ) /n )  n 2  and no mor e than O ( d (1 − 2 p ) − 2 log 2 ( n )) pairwise comparisons ar e r equested, on avera ge. If we repeat the algorithm with dif ferent initializations of the objects until a sufficient number of objects are ranked before an object is passed over , we can boost this constant probability to an arbitrarily high probability . Howe ver , in practice, we recommend running the algorithm just once to completion since we do not believ e passing ov er an object early on greatly af fects performance. 6 Empirical results In this section we present empirical results for both the error-free algorithm of Figure 1 and the robust algorithm of Figure 3. For the error-free algorithm, n = 100 points, representing the objects to be ranked, were uniformly at random simulated from the unit hypercube [0 , 1] d for d = 1 , 10 , 20 , . . . , 100 . The reference was simulated from the same distribution. For each value of d the experiment was repeated 25 times using a new simulation of points and the reference. Be- cause responses are error-free, exact identification of the ranking is guaranteed. The number of requested queries is plotted in Figure 4 with the lower bound of Theorem 1 for reference. The number of requested queries nev er exceeds twice the lo wer bound which agrees with the result of Theorem 3. The robust algorithm of Figure 3 was e valuated using a symmetric similarity matrix dataset a vailable at [21] whose ( i, j ) th entry , denoted s i,j , represents the human-judged similarity between audio signals i and j for all i 6 = j ∈ { 1 , . . . , 100 } . If we consider the k th row of this matrix, we can rank the other signals with respect to their similarity to the k th signal; we define q ( k ) i,j := { s k,i > s k,j } and y ( k ) i,j := 1 { q ( k ) i,j } . Since the similarities were deri ved from human subjects, the derived labels may be erroneous. Moreover , there is no possibility of repeating queries here and so the errors are persistent. The analysis of this dataset in [2] suggests that the relationship between signals can be well approximated by an embedding in 2 or 3 dimensions. W e used non-metric multidimensional scaling [5] to find an embedding of the signals: θ 1 , . . . , θ 100 ∈ R d for d = 2 and 3 . For each object 10 θ k , we use the embedding to deriv e pairwise comparison labels between all other objects as follo ws: ˜ y ( k ) i,j := 1 {|| θ k − θ i || < || θ k − θ j ||} , which can be considered as the best approximation to the labels y ( k ) i,j (defined above) in this embedding. The output of the robust sequential algorithm, which uses only a small fraction of the similarities, is denoted by ˆ y ( k ) i,j . W e set R = 15 using Theorem 6 as a rough guide. Using the popular Kendell-T au distance d ( y ( k ) , ˆ y ( k ) ) =  n 2  − 1 P i 2 da the number of requested queries to the k th object is upper bounded by the number of ambiguous queries 13 of the k th object. Then using the known mean and v ariance formulas for the binomial distrib ution E U  M n  = n − 1 X k =1 E U  B k +1  ≤ d 2 da e X k =2 B k +1 + n − 1 X k = d 2 da e +1 2 da k ≤ d 2 da e log 2 d 2 da e + 2 da log  n/ d 2 da e  ≤ d 2 da e log 2 n W e no w consider the case for a general distribution π . Enumerate the rankings of Σ n,d . Let N i denote the (random) number of requested queries needed by the algorithm to reconstruct the i th ranking. Note that the randomness of N i is only due to the randomization of the algorithm. Let π i denote the probability it assigns to the i th ranking as in Definition 1. Then E π [ M n ] = Q ( n,d ) X i =1 π i E [ N i ] . (8) Assume that the distrib ution ov er rankings is bounded above such that no ranking is overwhelmingly probable. Specifically , assume that the probability of any one ranking is upper bounded by c/Q ( n, d ) for some constant c > 1 that is independent of n . Under this bounded distribution assumption, E π [ M n ] is maximized by placing probability c/Q ( n, d ) on the k := Q ( n, d ) /c cells for which E [ N i ] is lar gest (we will assume k is an integer , but it is straightforward to extend the follo wing argument to the general case). Since the mass on these cells is equal, without loss of generality we may assume that E [ N i ] = µ , a common value on each, and we hav e E π [ M n ] = µ . For the remaining Q ( n, d ) − k cells we kno w that E [ N i ] ≥ d , since each cell is bounded by at least d hyperplanes/queries. Under these conditions, we can relate E π [ M n ] to E U [ M n ] as follows. First observe that E U [ M n ] = 1 Q ( n, d ) Q ( n,d ) X i =1 E [ N i ] ≥ k Q ( n, d ) µ + d Q ( n, d ) − k Q ( n, d ) , which implies E π [ M n ] = µ ≤ Q ( n,d ) k  E U [ M n ] − d Q ( n,d ) − k Q ( n,d )  = c  E U [ M n ] − d Q ( n,d ) − k Q ( n,d )  ≤ c E U [ M n ] . In words, the non-uniformity constant c > 1 scales the expected number of queries. Under A1-2 , for large n we ha ve E π [ M n ] = O ( c d log n ) . A.6 Proof of Theor em 4 Pr oof. Suppose q i,j is ambiguous. Let ˆ α be the frequency of Y i,j = 1 after R trials. Let E [ ˆ α ] = α . The majority v ote decision is correct if | α − ˆ α | ≤ 1 / 2 − p . By Chernof f ’ s bound, P ( | α − ˆ α | ≥ 1 / 2 − p ) ≤ 2 exp( − 2(1 / 2 − p ) 2 R ) . The result follo ws from the union bound over the total number of queries considered: n log 2 n (See Appendix A.1). A.7 Proof of Theor em 5 Suppose q i,j is ambiguous. Let S i,j denote the subset of Θ such that θ k ∈ S i,j if it is ranked between objects θ i and θ j (i.e. S i,j = { θ k ∈ Θ : θ i ≺ θ k ≺ θ j or θ j ≺ θ k ≺ θ i } ). Note that y i,j = y i,k = y k,j if and only if θ k ∈ S i,j . If we define E k i,j = 1 { Y i,k = 1 ∧ Y k,j = 1 } − 1 { Y i,k = 0 ∧ Y k,j = 0 } , where 1 is the indicator function, then for any subset T ⊂ Θ such that S i,j ⊂ T , the sign of the sum P θ k ∈ T E k i,j is a predictor of y i,j . In fact, with respect to just the random errors, E    P θ k ∈ T E k i,j    = | S i,j | (1 − 2 p ) . T o see this, without loss of generality let y i,j = 1 , then for 14 θ k ∈ S i,j E [ E k i,j ] = E  1 { Y i,k = 1 ∧ Y k,j = 1 } − 1 { Y i,k = 0 ∧ Y k,j = 0 }  = P ( Y i,k = 1 ∧ Y k,j = 1) − P ( Y i,k = 0 ∧ Y k,j = 0) = (1 − p ) 2 − p 2 = 1 − 2 p. If θ k / ∈ S i,j then it can be shown by a similar calculation that E [ E k i,j ] = 0 . T o identify S i,j we use the fact that if θ k ∈ S i,j then q i,k , q j,k , or both are also ambiguous simply because otherwise q i,j would not have been ambiguous in the first place (Figure 6 may be a useful aid to see this). While the con verse is false, Lemma 3 says that each of the six possible rankings of { θ i , θ j , θ k } are equally probable if they were uniformly at random chosen (thus partly justifying this e xplicit assumption in the theorem statement). It follo ws that if we define the subset T i,j ∈ Θ to be those objects θ k with the property that q i,k , q k,j , or both are ambiguous then the probability that θ k ∈ S i,j is at least 1 / 3 if θ k ⊂ T i,j . Y ou can con vince yourself of this using Figure 6. Moreover , E    P k ∈ T i,j E k i,j    ≥ | T i,j | (1 − 2 p ) / 3 which implies the sign of the sum P θ k ∈ T i,j E k i,j is a reliable predictor of q i,j ; just how reliable depends only on the size of T i,j . θ i θ j θ k θ j ≺ θ k ≺ θ i θ i ≺ θ k ≺ θ j θ i ≺ θ j ≺ θ k θ j ≺ θ i ≺ θ k θ k ≺ θ j ≺ θ i θ k ≺ θ i ≺ θ j Figure 6: Let q i,j be ambiguous. Object k will be informativ e to the majority vote of y i,j if the reference lies in the shaded region. There are six possible rankings and if q i,k , q k,j , or both are ambiguous then the probability that the reference is in the shaded region is at least 1 / 3 Fix R > 0 . Suppose q i,j is ambiguous and assume without loss of generality that y i,j = 1 . Giv en that E  P k ∈ T i,j E k i,j  ≥ | T i,j | (1 − 2 p ) / 3 from above, it follows from Hoeffding’ s inequality that the probability that P k ∈ T i,j E k i,j ≤ 0 is less than exp  − 2 9 (1 − 2 p ) 2 | T i,j |  . If only a subset of T i,j of size R is used in the sum then | T i,j | is replaced by R in the e xponent. This test is only performed when | T i,j | > R and clearly no more times than the number of queries considered to rank n objects in the full ranking: n log 2 n . Thus, all decisions using this test are correct with probability at least 1 − 2 n log 2 ( n ) exp  − 2 9 (1 − 2 p ) 2 R  . Only a subset of the n objects will be ranked and of those, 2 R + 1 times more queries will be requested than in the error-free case (two queries per object in T i,j ). Thus the robust algorithm will request no more than O ( Rd log n ) queries on a verage. T o determine the number of objects that are in the partial ranking, let Θ 0 ⊂ Θ denote the subset of objects that are ranked in the output partial ranking. Each θ k ∈ Θ 0 is associated with an index in the true full ranking and is denoted by σ ( θ k ) . That is, if σ ( θ k ) = 5 then it is ranked fifth in the full ranking but in the partial ranking could be ranked first, second, third, fourth, or fifth. Now imagine the real line with tick marks only at the integers 1 , . . . , n . F or each θ k ∈ Θ 0 place an R -ball around each θ k on these tick marks such that if σ ( θ k ) = 5 and R = 3 then 2 , . . . , 8 are covered by the ball around σ ( θ k ) and 1 and 9 , . . . , n are not. Then the union of the balls centered at the objects in Θ 0 cov er 1 , . . . , n . If this were not true then there would be an object θ j / ∈ Θ 0 with | S i,j | > R for all θ i ∈ Θ 0 . But S i,j ⊂ T i,j implies | T i,j | > R which implies j ∈ Θ 0 , a contradiction. Because at 15 least n/ (2 R + 1) R -balls are required to cov er 1 , . . . , n , at least this many objects are contained in Θ 0 . A.8 Proof of Lemma 5 Pr oof. Assume M ≤ n 3 R . If p m denotes the probability that the ( m + 1) st object is within R positions of one of the first m objects, giv en that none of the first m objects are within R positions of each other , then Rm n < p m ≤ 2 Rm n − m and P ( M = m ) ≥ m − 1 Y l =1  1 − 2 Rl n − l  Rm n . T aking the log we find log P ( M = m ) ≥ log Rm n + m − 1 X l =1 log  1 − 2 Rl n − l  ≥ log Rm n + ( m − 1) log  1 ( m − 1) m − 1 X l =1  1 − 2 Rl n − l  ≥ log Rm n + ( m − 1) log  1 − Rm n − m + 1  ≥ log Rm n + ( m − 1) log  1 − 3 Rm 2 n  ≥ log Rm n + ( m − 1)  − 3 log(2) Rm n  where the second line follows from Jensen’ s inequality , the fourth line follows from the fact that m ≤ n 3 R , and the last line follows from the fact that (1 − x ) ≥ exp( − 2 log(2) x ) for x ≤ 1 / 2 . W e conclude that P ( M = m ) ≥ R n m exp {− 3 log(2) R n m 2 } . Now if a = q n/R 6 log(2) we hav e P ( M ≥ a ) ≥ n/ (3 R ) − 1 X m = d a e R n m exp {− 3 log(2) R n m 2 } ≥ Z n/ (3 R ) a +1 R n x exp {− 3 log(2) R n x 2 } dx = 1 6 log(2)  e − ( √ 6 log(2) R/n +1) 2 / 2 − e − log(2) n/ (3 R )  where the second line follows from the fact that xe − αx 2 / 2 is monotonically decreasing for x ≥ p 1 /α . Note, P ( M ≥ q n/R 6 log(2) ) is greater than 1 100 for n/R ≥ 7 , and 1 10 for n/R ≥ 40 . Moreov er , as n/R → ∞ , P ( M ≥ q n/R 6 log(2) ) → 1 6 √ e log(2) . 16 A.9 Proof of Lemma 6 Pr oof. Enumerate the objects such that the first m are the objects ranked amongst themselves. Let y be the pairwise comparison label vector for σ and ˆ y be the corresponding vector for b σ . Then E [ d τ ( σ, b σ )] = m X k =2 k − 1 X l =1 1 { y l,k 6 = ˆ y l,k } + n X k = m +1 k − 1 X l =1 1 { y l,k 6 = ˆ y l,k } = n X k = m +1 k − 1 X l =1 1 { y l,k 6 = ˆ y l,k } ≤ n X k = m +1 k − 1 X l =1 P { Request q l,k | labels to q s ≤ m,t ≤ m } ≤ n X k = m +1 k − 1 X l =1 2 ad m 2 ≤ 2 ad m 2 ( n − m )( n + m + 1) 2 ≤ ad  ( n + 1) 2 m 2 − 1  . where the third line assumes that e very pairwise comparison that is ambiguous (that is, cannot be imputed using the knowledge gained from the first m objects) is incorrect. The fourth line follows from the application of Lemma 3 and Lemma 4. 17