The Expressive Power of Binary Submodular Functions

The Expressiv e P o w er of Binary Submo dular F unctions ∗ Stanisla v ˇ Zivn´ y Oxford Univ ersity Computing Lab oratory stanislav.zivny@comlab.ox.ac.uk Da vid A. Cohen Ro yal Hollo w a y , Univ ersity of London Departmen t of Computer Science dave@cs.rhul.ac.uk P eter G. Jeav ons Oxford Univ ersity Computing Lab oratory peter.jeavons@comlab.ox.ac.uk Abstract It has previously been an open problem whether all Boolean submodular functions can be decomp osed in to a sum of binary submodular functions ov er a p ossibly larger set of v ariables. This problem has been considered within sev eral differen t con texts in computer science, including computer vision, artificial in telligence, and pseudo-Boolean optimisation. Using a connection betw een the expressiv e p o wer of v alued constrain ts and certain algebraic prop erties of functions, w e answ er this question negatively . Our results ha ve several corollaries. First, w e c haracterise precisely which submo dular functions of arit y 4 can be expressed b y binary submo dular functions. Next, we identify a no vel class of submodular functions of arbitrary arities which can be expressed by binary submo dular functions, and therefore min- imised efficiently using a so-called expressibilit y reduction to the Min-Cut problem. More imp ortan tly , our results imply limitations on this kind of reduction and establish for the first time that it cannot b e used in general to minimise arbitrary submo dular functions. Finally , we refute a conjecture of Promislow and Y oung on the structure of the extreme rays of the cone of Bo olean submo dular functions. Keyw ords: Com binatorial optimisation, decomp osition of submodular functions, expressive p ow er, Gibbs energy minimisation, Mark ov Random Fields, min cut, multimorphisms, submo dular pseudo-Boolean min- imisation, submo dular polynomials, v alued constraint satisfaction problems. 1 In tro duction 1.1 Bac kground A function f : 2 V → R is called submo dular if for all S , T ⊆ V , f ( S ∩ T ) + f ( S ∪ T ) ≤ f ( S ) + f ( T ) . Submo dular functions are a key concept in op erational researc h and comb inatorial optimisation [ 39 , 38 , 48 , 47 , 17 , 33 , 27 ]. Examples include cut capacity functions, matroid rank functions, and entrop y functions. Submo dular functions are often considered as a discrete analogue of conv ex functions [ 36 ]. Both minimising and maximising submodular functions, p ossibly under some additional conditions, hav e b een considered extensively in the literature. Submo dular function maximisation is easily shown to b e NP- hard [ 47 ] since it generalises many standard NP-hard problems such as the maximum cut problem. In contrast, ∗ An earlier v ersion of some parts of the results of this article appeared in Pr o c e e dings of the 14th International Confer enc e on Principles and Pr actise of Constr aint Pr o gr amming (CP) , 2008, pp. 112–127, and in Oxford Universit y Computing Lab oratory T echnical Report CS-RR-08-08, June 2008. 1 the problem of minimising a submodular function ( SFM ) can be solved efficiently with only p olynomially man y oracle calls, either b y using the ellipsoid algorithm [ 20 , 21 ], or by using one of sev eral com binatorial algorithms that hav e been obtained in the last decade [ 46 , 26 , 24 , 25 , 40 , 28 ]. The time complexity of the fastest kno wn general algorithm for SFM is O ( n 6 + n 5 L ), where n is the n umber of v ariables and L is the time required to ev aluate the function [ 40 ]. The minimisation of submo dular functions on sets is equiv alent to the minimisation of submo dular func- tions on distributive lattices [ 47 ]. Krokhin and Larose hav e also studied the more general problem of min- imising submo dular functions on non-distributiv e lattices [ 34 ]. An imp ortan t and well-studied sub-problem of SFM is the minimisation of submo dular functions of b ounded arity ( SFM b ), also known as lo c al ly define d submo dular functions. In this scenario the submo dular function to be minimised is defined as the sum of a collection of functions whic h each dep end only on a b ounded num ber of v ariables. Locally defined optimisation problems o ccur in a v ariety of contexts: • In the con text of pseudo-Boolean optimisation, such problems in v olve the minimisation of Boolean p olynomials of bounded de gr e e [ 4 ]. • In the con text of artificial in telligence, they hav e been studied as value d c onstr aint satisfaction pr oblems ( V CSP ) [ 44 ], also known as soft or weighte d constraint satisfaction problems. • In the con text of computer vision, such problems are often formulated as Gibbs ener gy minimisation problems [ 19 ] or Markov R andom Fields [ 35 , 49 ]. W e will present our results primarily in the language of pseudo-Boolean optimisation. Hence an instance of SFM b with n v ariables will be represented as a polynomial in n Boolean v ariables, of some fixed b ounded degree. Ho wev er, we will also mention the consequences of our results for constrain t satisfaction problems and certain optimisation problems arising in computer vision. A general algorithm for SFM can alwa ys be used for the more restricted SFM b , but the special features of this more restricted problem sometimes allow more efficien t special-purp ose algorithms to b e used. (Note that w e are fo cusing on exact algorithms whic h find an optim um solution.) In particular, it has b een sho wn that certain cases can b e solved muc h more efficiently by reducing to the Min-Cut problem, that is, the problem of finding a minimum cut in a directed graph whic h includes a giv en source v ertex and excludes a giv en target vertex. F or example, it has b een kno wn since 1965 that the minimisation of quadr atic submodular p olynomials is equiv alent to finding a minim um cut in a corresp onding directed graph [ 23 , 4 ]. Hence quadratic submo dular p olynomials can b e minimised in O ( n 3 ) time, where n is the num b er of v ariables. A similar approach, using a reduction to Min-Cut , can b e used for any class of polynomials whic h can be decomp osed into a sum of quadratic submo dular polynomials, p erhaps with additional v ariables to b e minimised o v er. W e will say that a p olynomial that can be decomp osed in this wa y is expr essible by quadratic submodular p olynomials (see Section 1.1 ). The following classes of functions ha ve all b een shown to b e expressible in this wa y , o ver the past four decades: • p olynomials where all terms of degree 2 or more hav e negativ e co efficien ts (also known as negativ e- p ositiv e p olynomials) [ 43 ]; • cubic submodular p olynomials [ 2 ]; • { 0 , 1 } -v alued submo dular functions (also known as 2-monotone functions) [ 13 , 10 ]; • binary submodular functions o ver non-Bo olean domains [ 9 ] (also kno wn as Monge matrices [ 6 ]); • generalised 2-monotone functions [ 10 ]; • a class recently found b y ˇ Zivn´ y and Jeav ons [ 52 ] and independently by Zalesky [ 51 ]. All these classes of functions hav e b een sho wn to b e expressible by quadratic submodular p olynomials and hence can b e minimised in cubic time. This series of positive expressibility results naturally raises the follo wing question: 2 Problem 1. A r e all submo dular p olynomials expr essible by quadr atic submo dular p olynomials? Eac h of the ab o ve expressibilit y results was obtained by an ad-hoc construction, and no general technique has previously b een prop osed which is sufficien tly p o werful to address Problem 1 . 1.2 Con tributions Cohen et al. recen tly developed a no vel algebraic approach to characterising the expressive p ow er of v alued constrain ts in terms of certain algebraic prop erties of those constraints [ 7 ]. Using this systematic algebraic approach we are able to give a negativ e answ er to Problem 1 : we show that there are quartic submodular polynomials which are not expressible b y quadratic submodular polynomials. More precisely , w e characterise exactly which quartic submodular polynomials are expressible by quadratic submo dular polynomials and which are not. In addition, we sho w that any quartic submo dular polynomial is either expressible by quadratic submo dular p olynomials with only linearly many extra v ariables, or it is not expressible at all. On the wa y to establishing this result we show that tw o broad families of submo dular functions known as upp er fans and lower fans are all expressible b y binary submo dular functions. This provides a new class of submo dular p olynomials of all arities which are expressible by quadratic submo dular p olynomials and hence solv able efficien tly b y reduction to Min-Cut . W e use the expressibility of this family , and the existence of non-expressible functions, to refute a conjecture from [ 41 ] on the structure of the extreme rays of the cone of Bo olean submo dular functions, and suggest a more refined conjecture of our o wn. 1.3 Applications The concept of submo dularit y is important in a wide v ariety of fields within computer science; in this pap er w e briefly discuss tw o of these: artificial intelligence and computer vision. Our results can b e directly applied to b oth of these areas, as we show in Section 3.4 b elo w. Artificial In telligence A ma jor area of inv estigation in artificial in telligence is the Constr aint Satisfaction problem ( CSP ) [ 44 ]. A num b er of extensions hav e b een added to the basic CSP framework to deal with questions of optimisation, including semi-ring CSP s, v alued CSP s, soft CSP s and weigh ted CSP s. These extended frameworks can b e used to mo del a wide range of discrete optimisation problems [ 45 , 3 , 44 ], including standard problems suc h as Min-Cut , Max-Sa t , Max-Ones Sa t , Max-CSP [ 13 , 11 ], and Min- Cost Homomorphism [ 22 ]. The differences b et w een the v arious frameworks are not relev ant for our purposes, so we will simply fo cus on one very general framew ork, the v alued constraint satisfaction problem or V CSP . Informally , in the VCSP framew ork, an instance consists of a set of v ariables, a set of p ossible v alues for those v ariables, and a set of constrain ts. Each constrain t has an associated cost function which assigns a cost (or degree of violation) to every p ossible tuple of v alues for the v ariables in the scop e of the constraint. The goal is to find an assignment of v alues to all of the v ariables whic h has the minimum total cost. The class of constrain ts with submo dular cost functions is the only non-trivial tractable class of optimi- sation problems in the dichotom y classification of the Bo olean V CSP [ 11 ], and the only tractable class in the dichotom y classification of the Max-CSP problem for both 3-element sets [ 30 ] and arbitrary finite sets allo wing constan t (that is, fixed-v alue) constraints [ 14 ]. Cohen et al. sho wed that V CSP instances with submodular constrain ts ov er an arbitrary finite domain can be reduced to SFM [ 11 ], and hence can be solved in p olynomial time. This tractability result has since b een generalised to a wider class of v alued constrain ts ov er arbitrary finite domains kno wn as tournament- pair constraints [ 8 ]. An alternative approach to solving VCSP instances with b ounded-arit y submo dular constrain ts, based on linear programming, can b e found in [ 12 ]. Computer Vision Gibbs energy minimisation and Marko v Random Fields, play an important role in computer vision as they are applicable to a wide v ariety of vision problems, including image restoration, 3 stereo vision and motion tracking, image synthesis, image segmen tation, multi-camera scene reconstruction and medical imaging [ 32 ]. Reducing energy minimisation to the Min-Cut problem has recently become a v ery p opular approac h, leading to the redisco very of the prop ert y of submo dularit y [ 32 , 16 ], and sho wing that certain sp ecial classes of functions can b e minimised using graph cuts by introducing extra v ariables [ 42 , 31 ]. Our results b elo w characterise precisely which 4-ary submo dular functions can b e minimised using graph cuts in this w ay and which cannot. W e also pro vide a new class of submo dular functions of arbitrary arity whic h can b e minimised efficiently in this wa y . 2 Preliminaries In this section, w e in tro duce the basic definitions and the main to ols used throughout the pap er. 2.1 Cost functions and expressibilit y W e denote by R the set of all real n umbers together with (p ositiv e) infinit y . F or any fixed set D , a function φ from D n to R will b e called a c ost function on D of arity n . If the range of φ lies entirely within R , then φ is called a finite-value d cost function. If the range of φ is { 0 , ∞} , then φ can b e view ed as a predicate, or r elation , allowing just those tuples t ∈ D n for which φ ( t ) = 0. Cost functions can b e added and m ultiplied b y arbitrary real v alues, hence for an y giv en set of cost functions, Γ, we can define the con vex cone generated by Γ, as follows. Definition 2.1. F or any set of c ost functions Γ , the cone generated b y Γ , denote d Cone(Γ) , is define d by: Cone(Γ) = { α 1 φ 1 + · · · α r φ r | r ≥ 1; φ 1 , . . . , φ r ∈ Γ; α 1 , . . . α r ≥ 0 } . Definition 2.2. A c ost function φ of arity n is said to b e expressible by a set of c ost functions Γ if φ = min y 1 ,...,y j φ 0 ( x 1 , . . . , x n , y 1 , . . . , y j ) + κ , for some φ 0 ∈ Cone(Γ) and some c onstant κ . The variables y 1 , . . . , y j ar e c al le d extra (or hidden ) variables, and φ 0 is c al le d a gadget for φ over Γ . Note that in the sp ecial case of relations this notion of expressibilit y corresponds to the standard notion of expressibility using conjunction and existential quantification ( primitive p ositive formulas ) [ 5 ]. W e denote by h Γ i the expr essive p ower of Γ, whic h is the set of all cost functions expressible by Γ. It was shown in [ 7 ] that the expressiv e p o wer of a set of cost functions is determined b y certain algebraic prop erties of those cost functions called fractional p olymorphisms. F or the results of this pap er, w e will only need a certain subset of these algebraic prop erties, called multimorphisms [ 11 ]. These are defined in Definition 2.3 b elo w, whic h is illustrated in Figure 1 . The i -th component of a tuple t will be denoted by t [ i ]. Note that any op eration on a set D can b e extended to tuples ov er the set D in a standard w ay , as follows. F or an y function f : D k → D , and any collection of tuples t 1 , . . . , t k ∈ D n , define f ( t 1 , . . . , t k ) ∈ D n to b e the tuple h f ( t 1 [1] , . . . , t k [1]) , . . . , f ( t 1 [ n ] , . . . , t k [ n ]) i . Definition 2.3 ([ 11 ]) . L et F : D k → D k b e the function whose k -tuple of output values is given by the tuple of functions F = h f 1 , . . . , f k i , wher e e ach f i : D k → D . F or any n -ary c ost function φ , we say that F is a k -ary m ultimorphism of φ if, for al l t 1 , . . . , t k ∈ D n , k X i =1 φ ( t i ) ≥ k X i =1 φ ( f i ( t 1 , . . . , t k )) . F or an y set of cost functions, Γ, we will say that F is a m ultimorphism of Γ if F is a multimorphism of ev ery cost function in Γ. The set of all m ultimorphisms of Γ will be denoted Mul (Γ). Note that multimorphisms are preserv ed under expressibility . In other words, if F ∈ Mul (Γ), and φ ∈ h Γ i , then F ∈ Mul ( { φ } ) [ 11 , 7 ]. This has t w o imp ortan t corollaries. First, if h Γ 1 i = h Γ 2 i , then Mul (Γ 1 ) = Mul (Γ 2 ). Second, if there exists F ∈ Mul (Γ) such that F 6∈ Mul ( { φ } ), then φ is not expressible o ver Γ, that is, φ 6∈ h Γ i . 4 t 1 t 2 . . . t k t 0 1 = f 1 ( t 1 , . . . , t k ) t 0 2 = f 2 ( t 1 , . . . , t k ) . . . t 0 k = f k ( t 1 , . . . , t k ) t 1 [1] t 1 [2] . . . t 1 [ n ] t 2 [1] t 2 [2] . . . t 2 [ n ] . . . t k [1] t k [2] . . . t k [ n ] t 0 1 [1] t 0 1 [2] . . . t 0 1 [ n ] t 0 2 [1] t 0 2 [2] . . . t 0 2 [ n ] . . . t 0 k [1] t 0 k [2] . . . t 0 k [ n ] φ − → φ ( t 1 ) φ ( t 2 ) . . . φ ( t k )          k X i =1 φ ( t i ) ≥ φ − → φ ( t 0 1 ) φ ( t 0 2 ) . . . φ ( t 0 k )          k X i =1 φ ( t 0 i ) Figure 1: Inequalit y establishing F = h f 1 , . . . , f k i as a multimorphism of cost function φ (see Definition 2.3 ). 2.2 Lattices and submo dularit y Recall that L is a lattic e if L is a partially ordered set in which ev ery pair of elemen ts ( a, b ) has a unique suprem um (the least upp er b ound of a and b , called the join , denoted a ∨ b ) and a unique infim um (the greatest low er bound, called the me et , denoted a ∧ b ). F or an y lattice-ordered set D , a cost function φ : D n → R is called submo dular if for ev ery u, v ∈ D m , φ (min( u, v )) + φ (max( u, v )) ≤ φ ( u ) + φ ( v ) where b oth min and max are applied coordinate-wise on tuples u and v [ 39 ]. This standard definition can b e reform ulated very simply in terms of multimorphisms: φ is submo dular if h min , max i ∈ Mul ( { φ } ). Using results from [ 47 ] and [ 11 ], it can be shown that an y submo dular cost function φ can b e expressed as the sum of a finite-v alued submo dular cost function φ f in , and a submo dular relation φ rel , that is, φ = φ f in + φ rel . Moreo ver, it is kno wn that all submodular r elations are binary decomp osable [ 29 ], and hence expressible using only binary submo dular relations. Therefore, when considering which cost functions are express ible by binary submo dular cost functions, w e can restrict our attention to finite-value d cost functions without an y loss of generality . Next w e define some particular families of submo dular cost functions, first described in [ 41 ], which will turn out to pla y a central role in our analysis. Definition 2.4. L et L b e a lattic e. We define the fol lowing c ost functions on L : • F or any set F of p airwise inc omp ar able elements ( a 1 , . . . , a m ) ⊆ L , such that e ach p air of distinct elements ( a i , a j ) has the same le ast upp er b ound, W F , the fol lowing c ost function is c al le d an upp er fan : φ F ( x ) =      − 2 if x ≥ W F , − 1 if x 6≥ W F , but x ≥ a i for some i, 0 otherwise . • F or any set G of p airwise inc omp ar able elements ( a 1 , . . . , a m ) ⊆ L , such that e ach p air of distinct elements ( a i , a j ) has the same gr e atest lower b ound, V G , the fol lowing c ost function is c al le d a lo wer fan : φ G ( x ) =      − 2 if x ≤ V G, − 1 if x 6≤ V G , but x ≤ a i for some i, 0 otherwise . W e call a cost function a fan if it is either an upp er fan or a lo wer fan. It is not hard to show that all fans are submo dular [ 41 ]. Note that our definition of fans is sligh tly more general than the definition in [ 41 ]. In particular, w e allow the set F to b e empty , in which case the corresp onding upper fan φ F is a constant function. 5 2.3 Bo olean cost functions and polynomials In this pap er we will focus on problems ov er Boolean domains, that is, where D = { 0 , 1 } . An y cost function of arit y n can b e represen ted as a table of v alues of size D n . Moreo ver, a finite- v alued cost function φ : D n → R on a Boolean domain D = { 0 , 1 } can also b e represented as a unique p olynomial in n (Bo olean) v ariables with co efficien ts from R (such functions are sometimes called pseudo- Bo ole an functions [ 4 ]). Hence, in what follo ws, w e will often refer to a finite-v alued cost function on a Boolean domain and its corresponding p olynomial interc hangeably . F or polynomials o v er Bo olean v ariables there is a standard wa y to define derivatives of eac h order (see [ 4 ]). F or example, the second order deriv ativ e of a p olynomial p , with resp ect to the first tw o indices, denoted δ 1 , 2 ( x ), is defined as p (1 , 1 , x ) − p (1 , 0 , x ) − p (0 , 1 , x ) + p (0 , 0 , x ). Analogously for all other pairs of indices. It w as shown in [ 15 ] that a p olynomial p ( x 1 , . . . , x n ) ov er Bo olean v ariables x 1 , . . . , x n represen ts a submo dular cost function if and only if its second order deriv ativ es δ i,j ( x ) are non-p ositiv e for all 1 ≤ i < j ≤ n and all x ∈ D n − 2 . An immediate corollary is that a quadratic p olynomial represen ts a submo dular cost function if and only if the co efficien ts of all quadratic terms are non-p ositiv e. Note that a cost function is called sup ermo dular if all its second order deriv ativ es are non-negative. Clearly , f is submo dular if and only if − f is supermo dular. Cost functions which are both submodular and sup ermodular (in other words, all second order deriv atives are equal to zero) are called mo dular , and p olynomials corresp onding to mo dular cost functions are linear [ 4 ]. Example 2.5. F or any set of indic es I = { i 1 , . . . , i m } ⊆ { 1 , . . . , n } we c an define a c ost function φ I in n variables as fol lows: φ I ( x 1 , . . . , x n ) = ( − 1 if ( ∀ i ∈ I )( x i = 1) , 0 otherwise . The p olynomial r epr esentation of φ I is p ( x 1 , . . . , x n ) = − x i 1 . . . x i m , which is a p olynomial of de gr e e m . Note that it is str aightforwar d to verify that φ I is submo dular by che cking the se c ond or der derivatives of p . However, the function φ I is also expr essible by quadratic p olynomials, using a single extr a variable, y , as fol lows: φ I ( x 1 , . . . , x n ) = min y ∈{ 0 , 1 } {− y + y X i ∈ I (1 − x i ) } . We r emark that this is a sp e cial c ase of the expr essibility r esult for ne gative-p ositive p olynomials first obtaine d in [ 43 ]. Note that when D = { 0 , 1 } , the set D n with the product ordering is isomorphic to the lattice of all subsets of an n -element set ordered b y inclusion. Hence, a cost function on a Bo olean domain can b e view ed as a cost function defined on a lattice of subsets, and we can apply Definition 2.4 to identify certain Bo olean functions as upp er fans or lo wer fans, as the following example indicates. Example 2.6. L et F = { I 1 , . . . , I r } b e a set of subsets of { 1 , 2 , . . . , n } such that for al l i 6 = j we have I i 6⊆ I j and I i ∪ I j = S F . By Definition 2.4 , the c orr esp onding upp er fan function φ F has the fol lowing p olynomial r epr esentation: p ( x 1 , . . . , x n ) = ( r − 2) Y i ∈ S F x i − Y i ∈ I 1 x i − · · · − Y i ∈ I r x i . W e remark that any p ermutation of a set D gives rise to an automorphism of cost functions ov er D . In particular, for any cost function f on a Bo olean domain D , the dual of f is the corresponding cost function whic h results from exchanging the v alues 0 and 1 for all v ariables. In other words, if p is the p olynomial represen tation of f , then the dual of f is the cost function whose p olynomial represen tation is obtained from p by replacing all v ariables x with 1 − x . Observ e that, due to symmetry , taking the dual preserves submo dularit y and expressibility by binary submo dular cost functions. It is not hard to see that upp er fans are duals of lo wer fans and vice v ersa. 6 3 Results In this section, w e presen t our main results. First, we show that fans of all arities are expressible by binary submo dular cost functions. Next, we characterise the m ultimorphisms of binary submodular cost functions. Finally , combining these results together, we c haracterise precisely whic h 4-ary submo dular cost functions are expressible by binary submo dular cost functions. More imp ortan tly , w e show that some submo dular cost functions are not expressible b y binary submodular cost functions, and therefore cannot b e minimised using the Min-Cut problem via an expressibility reduction. Finally , we describ e some applications of these results to v alued constrain t satisfaction problems and certain optimisation problems arising in computer vision. 3.1 Expressibilit y of upp er fans and low er fans W e denote by Γ sub , n the set of all finite-v alued submodular cost functions of arit y at most n on a Bo olean domain D , and we set Γ sub = S n Γ sub , n . W e denote by Γ fans , n the set of all fans of arit y at most n on a Bo olean domain D , and we set Γ fans = S n Γ fans , n . Our next result sho ws that Γ fans ⊆ h Γ sub , 2 i . Theorem 3.1. Any fan on a Bo ole an domain D is expr essible by binary submo dular functions on D using at most 1 + b m/ 2 c extr a variables, wher e m is the de gr e e of its p olynomial r epr esentation. Pr o of. Since upp er fans are dual to low er fans, it is sufficient to establish the result for upp er fans only . Let F = { I 1 , . . . , I r } b e a set of subsets of { 1 , 2 , . . . , n } suc h that for all i 6 = j we hav e I i 6⊆ I j and I i ∪ I j = S F , and let φ F b e the corresp onding upp er fan, as specified by Definition 2.4 . The p olynomial represen tation of φ F , p ( x 1 , . . . , x n ), is given in Example 2.6 . The degree of p is equal to the total n umber of v ariables o ccurring in it, which will b e denoted m . Note that m = | S F | . If r = 0, then φ F is constant, so the result holds trivially . If r = 1, we hav e F = { I } , where I = { i 1 , . . . , i m } and the p olynomial represen tation of φ F is − 2 x i 1 x i 2 · · · x i m . In this case, it w as shown in Example 2.5 that φ F can b e expressed by quadratic functions using one extra v ariable, as follows: − 2 x i 1 x i 2 · · · x i m = min y ∈{ 0 , 1 } { 2 y (( m − 1) − X i ∈ I x i ) } . F or the case when r > 1, w e first note that an y i ∈ S F must b elong to all the elemen ts of F except for at most one (otherwise there would be tw o elements of F , sa y I i and I j , suc h that I i ∪ I j 6 = S F , whic h con tradicts the choice of F ). W e will say that t wo elements of S F are e quivalent if they o ccur in exactly the same elements of F , that is, i 1 , i 2 ∈ S F are equiv alent if i 1 ∈ I j ⇔ i 2 ∈ I j for all j ∈ { i, . . . , r } . Equiv alent elemen ts i 1 and i 2 of S F can b e merged b y replacing them with a single new elemen t. In the p olynomial representation of φ F this corresp onds to replacing the v ariables x i 1 and x i 2 with a single new v ariable, z , corresponding to their pro duct. Note that the num ber of equiv alence classes of size t wo or greater is at most b m/ 2 c . After completing all suc h merging, w e obtain a new set F 0 = { I 0 1 , . . . , I 0 r 0 } with the property that | I 0 i | = m 0 − 1 for every i , where m 0 = | S F 0 | is the size of the common join of an y I 0 i , I 0 j ∈ F 0 . This set has a corresp onding new upper fan, φ F 0 , ov er the new merged v ariables. T o complete the pro of we will construct a simple gadget for expressing φ F 0 , and show how to use this to obtain a gadget for expressing the original upp er fan φ F . Note that the sets I 0 i are subsets of S F 0 , each of size m 0 − 1. An y such subset is uniquely determined by its single missing elemen t. W e denote b y K the set of elemen ts o ccurring in al l sets I 0 i and by L the set of elemen ts which are missing from one of these subsets. Clearly , | K | + | L | = m 0 . W e claim that the follo wing p olynomial is a gadget for expressing φ 0 F : p 0 ( z 1 , . . . , z m 0 ) = min y ∈{ 0 , 1 } { y (2( m 0 − 1) − | L | − X i ∈ L z i − 2 X i ∈ K z i ) } . 7 T o establish this claim, w e will compute the v alue of p 0 , for eac h p ossible assignmen t to the v ariables z 1 , . . . , z m 0 . Denote b y k 0 the n um b er of 0s assigned to v ariables in K , and by l 0 the n um b er of 0s as- signed to v ariables in L . Then we ha ve: p 0 ( z 1 , . . . , z m 0 ) = min y ∈{ 0 , 1 } y (2 m 0 − 2 − | L | − X i ∈ L z i − 2 X i ∈ K z i ) = min y ∈{ 0 , 1 } y (2 m 0 − 2 − | L | − ( | L | − l 0 ) − 2( m 0 − | L | − k 0 ) = min y ∈{ 0 , 1 } y (2 m 0 − 2 − 2 | L | + l 0 − 2 m 0 + 2 | L | + 2 k 0 ) = min y ∈{ 0 , 1 } y ( − 2 + 2 k 0 + l 0 ) . Hence if k 0 = l 0 = 0, then p 0 tak es the v alue -2. If k 0 = 0 and l 0 = 1, then p 0 tak es the v alue -1. In all other cases (that is, k 0 > 0 or l 0 > 1), p 0 tak es the v alue 0. By Definition 2.4 , this means that p 0 is the (unique) p olynomial representation for φ F 0 . Note that p 0 uses just one extra v ariable, y . Finally , w e show ho w to obtain a gadget for the original upp er fan φ F , from the p olynomial p 0 . Each v ariable in p 0 represen ts an equiv alence class of elemen ts of S F , so it can b e replaced by a term consisting of the product of the v ariables in this equiv alence class. In this wa y we obtain a new polynomial ov er the original v ariables containing linear and negative quadratic terms together with negative higher order terms (cubic or abov e) corresponding to every equiv alence class with 2 or more elemen ts. Ho wev er, eac h of these higher order terms can itself b e expressed by a quadratic submo dular p olynomial, by in tro ducing a single extra v ariable, as shown in the case when r = 1, ab ov e. Therefore, com bining each of these p olynomials, the total num b er of new v ariables in tro duced is at most 1 + b m/ 2 c . Man y of the earlier expressibility results men tioned in Section 1.1 can be obtained as simple corollaries of Theorem 3.1 , as the follo wing examples indicate. Example 3.2. Any ne gative monomial − x 1 x 2 · · · x m is a p ositive multiple of an upp er fan, and the p ositive line ar monomial x 1 is e qual to − (1 − x 1 ) + 1 , so it is a p ositive multiple of a lower fan, plus a c onstant. Henc e, by The or em 3.1 , al l ne gative-p ositive submo dular p olynomials ar e expr essible by quadr atic submo dular p olynomials, as original ly shown in [ 43 ]. Example 3.3. Any cubic submo dular p olynomial c an b e expr esse d as a p ositive sum of upp er fans [ 41 ]. Henc e, by The or em 3.1 , al l cubic submo dular p olynomials ar e expr essible by quadr atic submo dular p olynomials, as original ly shown in [ 2 ]. Example 3.4. A Bo ole an c ost function φ is c al le d 2-monotone [ 13 ] if ther e exist two sets A, B ⊆ { 1 , . . . , n } such that φ ( x ) = 0 if A ⊆ x or x ⊆ B and φ ( x ) = 1 otherwise (wher e A ⊆ x me ans ∀ i ∈ A, x [ i ] = 1 and x ⊆ B me ans ∀ i 6∈ B , x [ i ] = 0 ). It was shown in [ 10 , Pr op osition 2.9] that a 2-value d Bo ole an c ost function is 2-monotone if and only if it is submo dular. 1 F or any 2-monotone c ost function define d by the sets of indic es A and B , it is str aightforwar d to che ck that φ = min y ∈{ 0 , 1 } y (1 + φ F / 2) + (1 − y )(1 + φ G / 2) wher e φ F is the upp er fan define d by F = { A } and φ G is the lower fan define d by G = { B } . Note that the function y φ F is an upp er fan, and the function (1 − y ) φ G is a lower fan. Henc e, by The or em 3.1 , al l 2-monotone p olynomials ar e expr essible by quadr atic submo dular p olynomials, and solvable by r e duction to Min-Cut , as original ly shown in [ 13 ]. Ho wev er, Theorem 3.1 also provides man y new functions of all arities which hav e not previously b een sho wn to b e expressible by quadratic submodular functions, as the follo wing example indicates. Example 3.5. The function 2 x 1 x 2 x 3 x 4 − x 1 x 2 x 3 − x 1 x 2 x 4 − x 1 x 3 x 4 − x 2 x 3 x 4 b elongs to Γ fans , 4 , but do es not b elong to any class of submo dular functions which has pr eviously b e en shown to b e expr essible by quadr atic submo dular functions. In p articular, it do es not b elong to the class Γ new identifie d in [ 52 , 51 ]. 1 In fact, [ 10 ] studied sup ermo dular cost functions, but as f is sup ermodular if and only if − f is submo dular, the results translate easily . 8 3.2 Characterising Mul (Γ sub , 2 ) Since we hav e seen that a cost function can only b e expressed b y a giv en set of cost functions if it has the same multimorphisms, we now inv estigate the multimorphisms of Γ sub , 2 . A function F : D k → D k is called c onservative if, for eac h p ossible choice of x 1 , . . . , x k , the tuple F ( x 1 , . . . , x k ) contains the same m ulti-set of v alues, x 1 , . . . , x k (in some order). F or an y tw o tuples x = h x 1 , . . . , x k i and y = h y 1 , . . . , y k i ov er D , we denote b y H ( x , y ) the Hamming distanc e b et ween x and y , whic h is the n umber of p ositions at which the corresp onding v alues are different. Theorem 3.6. F or any Bo ole an domain D , and any F : D k → D k , the fol lowing ar e e quivalent: 1. F ∈ Mul (Γ sub , 2 ) . 2. F ∈ Mul (Γ ∞ sub , 2 ) , wher e Γ ∞ sub , 2 denotes the set of binary submo dular c ost functions taking finite or infinite values. 3. F is c onservative and Hamming distanc e non-incr e asing. Pr o of. First w e consider unary cost functions. All unary cost functions on a Boolean domain are easily sho wn to be submodular. Also, any conserv ative function F : D k → D k is clearly a m ultimorphism of any unary cost function, since it merely p erm utes its arguments. F or any d ∈ D and c ∈ R , define the unary cost function µ d c as follows: µ d c ( x ) = ( c if x = d, 0 if x 6 = d. Let F : D k → D k b e a non-conserv ative function. In that case, there are u 1 , . . . , u k , v 1 , . . . , v k ∈ D suc h that F ( u 1 , . . . , u k ) = h v 1 , . . . , v k i and there is i such that v i o ccurs more often in h v 1 , . . . , v k i than in h u 1 , . . . , u k i . It is simple to chec k that F is not a multimorphism of the unary cost function µ v i 1 . Hence any F ∈ Mul (Γ sub , 2 ) must b e conserv ative. By the same argumen t, an y F ∈ Mul (Γ ∞ sub , 2 ) must b e conserv ative. F or any c ∈ R , define the binary cost functions λ c and χ c as follows: λ c ( x, y ) = ( c if x = 0 and y = 1 , 0 otherwise . χ c ( x, y ) = ( c if x 6 = y , 0 otherwise. Note that χ c ( x, y ) = λ c ( x, y ) + λ c ( y , x ). By a simple case analysis, it is straigh tforw ard to chec k that any binary submodular cost function on a Bo olean domain can b e expressed b y binary functions of the form λ c , with c > 0 together with unary cost functions of the form µ d c . W e observe that when c < ∞ , λ c ( x, y ) = ( χ c ( x, y ) + µ 0 c ( x ) + µ 1 c ( y ) − c ) / 2, so λ c can b e expressed by functions of the form χ c together with unary cost functions of the form µ d c . Hence, since expressibility preserv es m ultimorphisms, Mul (Γ sub , 2 ) = Mul ( { χ c | c ∈ R , c > 0 } ) ∩ Mul ( { µ d c | c ∈ R , d ∈ D } ). No w let u , v ∈ D k , and consider the multimorphism inequalit y , as giv en in Definition 2.3 , for the case where t i = h u [ i ] , v [ i ] i , for i = 1 , . . . , k . By Definition 2.3 , for an y c > 0, F is a m ultimorphism of χ c if and only if the follo wing holds for all choices of u and v : H ( u , v ) ≥ H ( F ( u ) , F ( v )) . This prov es that the m ultimorphisms of Γ sub , 2 are precisely the conserv ative functions which are also Hamming distance non-increasing. Since Γ sub , 2 ⊆ Γ ∞ sub , 2 , we know that Mul (Γ ∞ sub , 2 ) ⊆ Mul (Γ sub , 2 ). Therefore, in order to complete the pro of it is enough to show that every conserv ative and Hamming distance non-increasing function F is a m ultimorphism of λ ∞ . 9 F or any u , v ∈ { 0 , 1 } k , the Hamming distance H ( u , v ) is equal to the symmetric difference of the sets of p ositions where u and v tak e the v alue 1. Hence, for tuples u and v containing some fixed n umber of 1s, the minim um Hamming distance o ccurs precisely when one of these sets of p ositions is contained in the other. No w consider again the m ultimorphism inequalit y , as giv en in Definition 2.3 , for the case where t i = h u [ i ] , v [ i ] i , for i = 1 , . . . , k . If there is an y p osition i where u [ i ] = 0 and v [ i ] = 1, then λ ∞ ( t i ) = ∞ , so the m ultimorphism inequalit y is trivially satisfied. If there is no suc h position, then the set of p ositions where v tak es the v alue 1 is contained in the set of p ositions where u tak es the v alue 1, so H ( u , v ) takes its minimum p ossible v alue o ver all reorderings of u and v . Hence if F is conserv ative, then H ( u , v ) ≤ H ( F ( u ) , F ( v )), and if F is Hamming distance non-increasing, w e ha ve H ( u , v ) = H ( F ( u ) , F ( v )). But this implies that the set of positions where F ( v ) tak es the v alue 1 is con tained in the set of p ositions where F ( u ) tak es the v alue 1. By definition of λ ∞ , this implies that both sides of the m ultimorphism inequality are zero, so F is a m ultimorphism of λ ∞ . 3.3 Non-expressibilit y of Γ sub o v er Γ sub , 2 Consider the (carefully chosen) function F sep : { 0 , 1 } 5 → { 0 , 1 } 5 defined in Figure 2 . W e will show in this section that this particular function can b e used to characterise all the submo dular functions of arity 4 which are expressible by binary submodular functions on a Bo olean domain, and hence show that some submo dular functions are not expressible. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 x 0 0 0 0 1 1 1 1 0 0 0 0 1 1 1 1 0 0 0 0 1 1 1 1 0 0 0 0 1 1 1 1 0 0 1 1 0 0 1 1 0 0 1 1 0 0 1 1 0 0 1 1 0 0 1 1 0 0 1 1 0 0 1 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 F sep ( x ) 0 0 0 0 0 0 1 1 0 0 0 1 0 0 1 1 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 0 0 0 1 0 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 Figure 2: Definition of F sep . Prop osition 3.7. F sep is c onservative and Hamming distanc e non-incr e asing. Pr o of. Straigh tforward exhaustive verification. Theorem 3.8. F or any function f ∈ Γ sub , 4 the fol lowing ar e e quivalent: 1. f ∈ h Γ sub , 2 i ; 2. F sep ∈ Mul ( { f } ) ; 3. f ∈ Cone(Γ fans , 4 ) . Pr o of. Prop osition 3.7 and Theorem 3.6 imply that F sep is a multimorphism of an y binary submo dular function on a Boolean domain. Hence having F sep as a multimorphism is a necessary condition for an y submo dular cost function on a Bo olean domain to b e expressible b y binary submo dular cost functions. W e will no w complete the pro of b y sho wing that for 4-ary submodular cost functions on a Bo olean domain ha ving F sep as a multimorphism is also sufficient to ensure expressibilit y b y binary cost functions. W e consider the complete set of inequalities on the v alues of a 4-ary cost function resulting from having the m ultimorphism F sep , as sp ecified in Definition 2.3 . Out of 16 5 suc h inequalities, there are 4635 whic h are distinct. After removing from these all those whic h are equal to the sum of t w o others, we obtain a 10 system of just 30 inequalities which must be satisfied by an y 4-ary submo dular cost function whic h has the m ultimorphism F sep . Using the double description metho d 2 [ 37 ] we obtain from these 30 inequalities an equiv alent set of 31 extreme rays which generate the same p olyhedral cone of cost functions. These extreme ra ys all corresp ond to fans or sums of fans, and hence are expressible ov er Γ sub , 2 , by Theorem 3.1 . It follows that any cost function in this cone of functions is also expressible ov er Γ sub , 2 . Next we show that there are indeed 4-ary submo dular cost functions whic h do not ha ve F sep as a multi- morphism and therefore are not expressible by binary submo dular cost functions. Definition 3.9. F or any Bo ole an tuple t of arity 4 c ontaining exactly 2 ones and two zer os, we define the 4-ary c ost function θ t as fol lows: θ t ( x 1 , x 2 , x 3 , x 4 ) =      − 1 if ( x 1 , x 2 , x 3 , x 4 ) = (1 , 1 , 1 , 1) or (0 , 0 , 0 , 0) , 1 if ( x 1 , x 2 , x 3 , x 4 ) = t, 0 otherwise. Cost functions of the form θ t w ere introduced in [ 41 ], where they are called quasi-inde c omp osable functions. W e denote by Γ qin the set of all (six) quasi-indecomp osable cost functions of arit y 4. It is straigh tforward to c heck that they are submodular, but the next result shows that they are not expressible b y binary submo dular functions. Prop osition 3.10. F or al l θ ∈ Γ qin , F sep 6∈ Mul ( { θ } ) . Pr o of. The table in Figure 3 shows that F sep 6∈ Mul ( { θ (1 , 1 , 0 , 0) } ). Perm uting the columns appropriately establishes the result for all other θ ∈ Γ qin . F sep 1 0 1 0 1 0 0 1 0 1 0 1 0 1 1 0 0 0 1 1 0 0 1 0 0 0 0 1 1 1 0 0 1 0 1 1 0 1 1 1 θ (1 , 1 , 0 , 0) − → 0 0 0 0 0            P = 0 θ (1 , 1 , 0 , 0) − → 0 0 1 0 0            P = 1 Figure 3: F sep 6∈ Mul ( { θ (1 , 1 , 0 , 0) } ) . Corollary 3.11. F or al l θ ∈ Γ qin , θ 6∈ h Γ sub , 2 i . Pr o of. By Theorem 3.8 and Prop osition 3.10 . Are there any other 4-ary submodular cost functions which are not expressible ov er Γ sub , 2 ? Promislow and Y oung characterised the extreme ra ys of the cone of all 4-ary submo dular 3 cost functions and established that Γ sub , 4 = Cone(Γ fans , 4 ∪ Γ qin ) – see Theorem 5.2 of [ 41 ]. Hence the results in this section characterise the expressibilit y of all 4-ary submodular functions. Promislo w and Y oung conjectured that for k 6 = 4, all extreme rays of Γ sub , k are fans [ 41 ]. How ever, if this conjecture were true it w ould imply that all submo dular functions of arit y 5 and ab o v e w ere expressible b y binary submo dular functions, by Theorem 3.1 . This is clearly not the case, because inexpressible cost 2 As implemented, for example, by the program Skeleton av ailable from http://www.uic.nnov.ru/~zny/skeleton/ 3 In fact, [ 41 ] studied sup ermo dular cost functions, but as f is sup ermodular if and only if − f is submo dular, the results translate easily . 11 functions such as those identified in Corollary 3.11 can b e extended to larger arities (e.g., by adding dummy argumen ts) and remain inexpressible. Hence our results refute this conjecture. How ev er, w e suggest that this conjecture can be refined to a similar statemen t concerning just those submo dular functions which are expressible by binary submo dular functions, as follows: Conjecture 3.12. F or al l k , Γ sub , k ∩ h Γ sub , 2 i = Cone(Γ fans , k ) . This conjecture was previously known to b e true for k ≤ 3 [ 41 ]; Theorem 3.8 confirms that it holds for k = 4. Next we show that w e can test efficiently whether a submo dular p olynomial of degree 4 is expressible by quadratic submo dular polynomials. Definition 3.13. L et p ( x 1 , x 2 , x 3 , x 4 ) b e the p olynomial r epr esentation of a 4-ary submo dular c ost function f . We denote by a I the c o efficient of the term Q i ∈ I x i . We say that f satisfies c ondition Sep if for e ach { i, j } , { k , l } ⊂ { 1 , 2 , 3 , 4 } , with i, j, k , l distinct, we have a { i,j } + a { k,l } + a { i,j,k } + a { i,j,l } ≤ 0 . Theorem 3.14. F or any f ∈ Γ sub , 4 , the fol lowing ar e e quivalent: 1. f ∈ h Γ sub , 2 i 2. f satisfies c ondition Sep. Pr o of. As in the proof of Theorem 3.8 , w e can construct a set of 30 inequalities corresp onding to the multi- morphism F sep . Each of these inequalities on the v alues of a cost function can b e translated into inequalities on the coefficients of the corresp onding polynomial representation. 24 of them imp ose the condition of sub- mo dularit y , and the remaining 6 inequalities imp ose condition Sep. Hence a submo dular cost function of arit y 4 has the multimorphism F sep if and only if its p olynomial represen tation satisfies condition Sep. The result then follows from Theorem 3.8 . Corollary 3.15. Given a submo dular p olynomial p of de gr e e 4, c ondition Sep c an b e use d to test in p olynomial time whether p is expr essible by quadr atic submo dular p olynomials. In contrast to this result, it is known that the recognition problem for submodular polynomials of degree 4 is co-NP-complete [ 18 ]. Given an arbitrary p olynomial of degree 4, condition Sep recognises expressible p olynomials under the assumption that the p olynomial is submo dular . One might hop e that submodular p olynomials whic h are expressible b y quadratic submodular p olynomials w ould b e recognisable in p olynomial time. Unfortunately , this is not the case. In fact, as all polynomials of degree 4 used in the reduction given in [ 18 ] satisfy condition Sep, the original reduction from [ 18 ] pro ves the follo wing: Prop osition 3.16. Given an arbitr ary p olynomial p of de gr e e 4, it is c o-NP-c omplete to test whether p is a submo dular p olynomial which is expr essible by quadr atic submo dular p olynomials. 3.4 Applications As mentioned ab o ve, testing submo dularit y is co-NP-complete ev en for polynomials of degree 4 [ 18 ]. How ev er, for man y of the optimisation problems arising in practice, testing for submo dularit y is not an issue b ecause the function to b e minimised is presen ted as a sum of functions of b ounded arity . In suc h cases, eac h of the b ounded-arit y sub-functions can b e tested for submo dularity in constant time. F or example, in constrain t satisfaction problems and computer vision, eac h instance is sp ecified as a sum of b ounded-arity functions and these can b e independently tested for submo dularit y . The recognition of submo dularit y only b ecomes co-NP-complete when a function is presen ted without a fixed decomposition into sub-functions of this kind. 12 Artificial In telligence First we formally define v alued constrain t satisfaction problems [ 45 , 3 , 44 ]. Definition 3.17. An instanc e P of VCSP is a triple h V , D , C i , wher e V is a finite set of v ariables , which ar e to b e assigne d values fr om the set D , and C is a set of v alued constraints . Each c ∈ C is a p air c = h σ, φ i , wher e σ is a tuple of variables of length | σ | , c al le d the scop e of c , and φ : D | σ | → R is a c ost function. An assignmen t for the instanc e P is a mapping s fr om V to D . The cost of an assignment s is define d as fol lows: C ost P ( s ) = X hh v 1 ,v 2 ,...,v m i ,φ i∈C φ ( h s ( v 1 ) , s ( v 2 ) , . . . , s ( v m ) i ) . A solution to P is an assignment with minimum c ost. No w w e show ho w our results can be applied in this framew ork. Corollary 3.18 (of Theorem 3.1 ) . VCSP (Γ fans ) is solvable in O (( n + k ) 3 ) time, wher e wher e n is the numb er of variables and k is the numb er of higher-or der (ternary and ab ove) c onstr aints. Moreo ver, as shown ab o ve, V CSP (Γ fans , 4 ) is the maximal class in VCSP (Γ sub , 4 ) which can b e solved by reduction to Min-Cut in this w ay . Cohen et al. [ 7 ] show ed that if a cost function φ of arity k is expressible by some set of cost functions o ver Γ, then φ is expressible by Γ using at most 2 2 k extra v ariables. Our results show that only O ( k ) extra v ariables are needed to express an y cost function from Γ fans , k b y Γ sub , 2 . Therefore, an instance of V CSP (Γ fans ) needs only linearly man y (in the num ber of higher-order constraints) extra v ariables, where the linear factor is prop ortional to the maximum arity of the constraints. In particular, an instance of VCSP (Γ sub , 4 ) is either reducible to Min-Cut with only linearly many extra v ariables, 4 or is not reducible at all. Computer Vision In computer vision, man y problems can b e naturally formulated in terms of energy minimisation where the energy function, o ver a set of v ariables { x v } v ∈ V , has the follo wing form: E ( x ) = c 0 + X v ∈ V c v ( x v ) + X h u,v i∈ V × V c uv ( x u , x v ) + . . . Set V usually corresp onds to pixels, x v denotes the lab el of of pixel v ∈ V which must belong to a finite domain D . The constan t term of the energy is c 0 , the unary terms c v ( · ) encode data p enalt y functions, the pairwise terms c uv ( · , · ) are interaction potentials, and so on. F unctions of arity 3 and ab ov e are also called higher-order cliques. This energy is often deriv ed in the con text of Markov R andom Fields [ 19 , 1 ]: a minim um of E corresp onds to a maximum a-p osteriori (MAP) lab elling x [ 35 , 49 ]. It is straigh tforward that this is equiv alen t to V CSP . See [ 50 ] for a surv ey on the connection b et w een computer vision and constrain t satisfaction problems. Therefore, for energy minimisation ov er Bo olean v ariables we get the following: Corollary 3.19 (of Theorem 3.1 ) . Ener gy minimisation, wher e e ach term of the ener gy function b elongs to Γ fans , is solvable in O (( n + k ) 3 ) time, wher e wher e n is the numb er of variables (pixels) and k is the numb er of higher-or der (ternary and ab ove) terms in the ener gy function. Note that any v ariable o ver a non-Bo olean domain D = { 0 , 1 , . . . , d − 1 } of size d can b e enco ded by d − 1 Bo olean v ariables. One suc h enco ding is the follo wing: en ( i ) = 0 d − i − 1 1 i . W e replace each v ari- able with d − 1 new Bo olean v ariables and imp ose a (submo dular) relation on these new v ariables which ensures that they only tak e v alues in the range of the enco ding function en . Note that en (max( a, b )) = max( en ( a ) , en ( b )) and en (min( a, b )) = min( en ( a ) , en ( b )), so this enco ding preserves submo dularity . Observe that an y submo dularit y-preserving encoding of a non-Boolean v ariable b y Bo olean v ariables needs at least O ( d ) v ariables. Ho wev er, for practical purp oses, sub classes of non-Bo olean submodular functions whic h can b e enco ded by Bo olean submo dular functions with fewer v ariables hav e been studied, as well as approximation algorithms for these problems [ 42 , 31 ]. 4 Optimal (in the num ber of extra v ariables) gadgets for cost functions from Γ fans , 4 were shown in [ 53 ]. 13 Ac kno wledgemen ts The authors w ould lik e to thank Martin Coop er for fruitful discussions on submodular functions and in partic- ular for help with the proof of Theorem 3.1 . Stanislav ˇ Zivn´ y w ould lik e to thank Philip T orr and his computer vision group, and T om´ a ˇ s W erner for clarifying the connection b etw een constrain t satisfaction problems and computer vision. Stanisla v ˇ Zivn´ y gratefully ackno wledges the supp ort of EPSR C gran t EP/F01161X/1. References [1] Besag, J.: On the statistical analysis of dirty pictures. Journal of the Roy al Statistical So ciet y , Series B 48 (3) (1986) 259–302 13 [2] Billionet, A., Minoux, M.: Maximizing a supermo dular pseudo-b oolean function: a p olynomial algorithm for cubic functions. Discrete Applied Mathematics 12 (1985) 1–11 2 , 8 [3] Bistarelli, S., F argier, H., Montanari, U., Rossi, F., Schiex, T., V erfaillie, G.: Semiring-based CSPs and v alued CSPs: F rameworks, prop erties, and comparison. Constraints 4 (1999) 199–240 3 , 13 [4] Boros, E., Hammer, P .L.: Pseudo-bo olean optimization. Discrete Applied Mathematics 123 (1-3) (2002) 155–225 2 , 6 [5] Bulatov, A., Krokhin, A., Jeav ons, P .: Classifying the complexity of constraints using finite algebras. SIAM Journal on Computing 34 (3) (2005) 720–742 4 [6] Burk ard, R., Klinz, B., Rudolf, R.: P ersp ectives of Monge prop erties in optimization. Discrete Applied Mathematics 70 (1996) 95–161 2 [7] Cohen, D., Co op er, M., Jea vons, P .: An algebraic c haracterisation of complexity for v alued constraints. In: CP’06. V olume 4204 of LNCS. (2006) 107–121 3 , 4 , 13 [8] Cohen, D., Co oper, M., Jeav ons, P .: Generalising submo dularit y and Horn clauses: T ractable optimiza- tion problems defined b y tournamen t pair m ultimorphisms. Theoretical Computer Science 401 (2008) 36–51 3 [9] Cohen, D., Co op er, M., Jeav ons, P ., Krokhin, A.: A maximal tractable class of soft constrain ts. Journal of Artificial Intelligence Research 22 (2004) 1–22 2 [10] Cohen, D., Co oper, M., Jea vons, P ., Krokhin, A.: Sup ermodular functions and the complexity of Max-CSP. Discrete Applied Mathematics 149 (2005) 53–72 2 , 8 [11] Cohen, D., Co oper, M., Jeav ons, P ., Krokhin, A.: The complexity of soft constraint satisfaction. Artificial In telligence 170 (2006) 983–1016 3 , 4 , 5 [12] Co oper, M.C.: Minimization of lo cally defined submo dular functions b y optimal soft arc consistency . Constrain ts 13 (2008) 3 [13] Creignou, N., Khanna, S., Sudan, M.: Complexity Classification of Boolean Constrain t Satisfaction Problems. V olume 7 of SIAM Monographs on Discrete Mathematics and Applications. SIAM (2001) 2 , 3 , 8 [14] Deineko, V., Jonsson, P ., Klasson, M., Krokhin, A.: The approximabilit y of Max CSP with fixed-v alue constrain ts. Journal of the ACM 55 (4) (2008) 3 [15] Fisher, M., Nemhauser, G., W olsey , L.: An analysis of approximations for maximizing submodular set functions-I. Mathematical Programming 14 (1978) 265–294 6 14 [16] F reedman, D., Drineas, P .: Energy minimization via graph cuts: Settling what is p ossible. In: CVPR’05, IEEE Computer So ciet y (2005) 939–946 4 [17] F ujishige, S.: Submo dular F unctions and Optimization. 2nd edn. V olume 58 of Annals of Discrete Mathematics. North-Holland, Amsterdam (2005) 1 [18] Gallo, G., Simeone, B.: On the supermo dular knapsac k problem. Mathematical Programming 45 (1988) 295–309 12 [19] Geman, S., Geman, D.: Sto c hastic Relaxation, Gibbs Distributions, and the Ba yesian Restoration of Images. IEEE Transactions on Pattern Analysis and Mac hine In telligence 6 (1984) 721–741 2 , 13 [20] Gr¨ otschel, M., Lo v asz, L., Sc hrijver, A.: The ellipsoid metho d and its consequences in combinatorial optimization. Com binatorica 1 (1981) 169–198 2 [21] Gr¨ otschel, M., Lo v asz, L., Sc hrijver, A.: Geometric Algorithms and Combinatorial Optimization. V ol- ume 2 of Algorithms and Com binatorics. Springer-V erlag (1988) 2 [22] Gutin, G., Rafiey , A., Y eo, A., Tso, M.: Lev el of repair analysis and minim um cost homomorphisms of graphs. Discrete Applied Mathematics 154 (2006) 881–889 3 [23] Hammer, P .L.: Some net w ork flo w problems solv ed with pseudo-b oolean programming. Op erations Researc h 13 (1965) 388–399 2 [24] Iwata, S.: A fully combinatorial algorithm for submo dular function minimization. Journal of Com bina- torial Theory , Series B 84 (2) (2002) 203–212 2 [25] Iwata, S.: A faster scaling algorithm for minimizing submodular functions. SIAM Journal on Computing 32 (4) (2003) 833–840 2 [26] Iwata, S., Fleisc her, L., F ujishige, S.: A com binatorial, strongly p olynomial-time algorithm for minimiz- ing submo dular functions. Journal of the ACM 48 (2001) 761–777 2 [27] Iwata, S.: Submo dular function minimization. Mathematical Programming 112 (2008) 45–64 1 [28] Iwata, S., Orlin, J.B.: A simple combinatorial algorithm for submo dular function minimization. In: SOD A’09. (2009) 2 [29] Jeav ons, P ., Cohen, D., Co op er, M.: Constraints, consistency and closure. Artificial Intelligence 101 (1– 2) (1998) 251–265 5 [30] Jonsson, P ., Klasson, M., Krokhin, A.: The approximabilit y of three-v alued MAX CSP. SIAM Journal on Computing 35 (6) (2006) 1329–1349 3 [31] Kohli, P ., Ladick´ y, L., T orr, P .: Graph Cuts for Minimizing Robust Higher Order Potentials. T echnical rep ort, Oxford Brookes Universit y (2008) 4 , 13 [32] Kolmogorov, V., Zabih, R.: What energy functions can b e minimized via graph cuts? IEEE T ransactions on Pattern Analysis and Mac hine In telligence 26 (2) (2004) 147–159 4 [33] Korte, B., Vygen, J.: Com binatorial Optimization. 4th edn. V olume 21 of Algorithms and Combinatorics. Springer-V erlag (2007) 1 [34] Krokhin, A., Larose, B.: Maximizing sup ermo dular functions on product lattices, with application to maxim um constrain t satisfaction. SIAM Journal on Discrete Mathematics 22 (1) (2008) 312–328 2 [35] Lauritzen, S.L.: Graphical Mo dels. Oxford Universit y Press (1996) 2 , 13 15 [36] Lov´ asz, L.: Submo dular functions and conv exity . In Bachem, A., Gr¨ otschel, M., Korte, B., eds.: Mathematical Programming – The State of the Art, Berlin, Springer-V erlag (1983) 235–257 1 [37] Motzkin, T., Raiffa, H., Thompson, G., Thrall, R.: The double description method. In Kuhn, H.W., T uck er, A.W., eds.: Contributions to the Theory of Games. V olume 2. Princeton Universit y Press (1953) 51–73 11 [38] Naray anan, H.: Submo dular F unctions and Electrical Netw orks. North-Holland, Amsterdam (1997) 1 [39] Nemhauser, G., W olsey , L.: In teger and Combinatorial Optimization. John Wiley & Sons (1988) 1 , 5 [40] Orlin, J.B.: A faster strongly p olynomial time algorithm for submo dular function minimization. In: IPCO’07. V olume 4513 of LNCS. (2007) 240–251 2 [41] Promislow, S., Y oung, V.: Sup ermo dular functions on finite lattices. Order 22 (4) (2005) 389–413 3 , 5 , 8 , 11 , 12 [42] Ramalingam, S., Kohli, P ., Alahari, K., T orr, P .: Exact Inference in Multi-label CRFs with Higher Order Cliques. In: CVPR’08, IEEE Computer So ciet y (2008) 4 , 13 [43] Rhys, J.: A selection problem of shared fixed costs and net work flows. Managemen t Science 17 (3) (1970) 200–207 2 , 6 , 8 [44] Rossi, F., v an Beek, P ., W alsh, T., eds.: The Handb o ok of Constrain t Programming. Elsevier (2006) 2 , 3 , 13 [45] Schiex, T., F argier, H., V erfaillie, G.: V alued constrain t satisfaction problems: hard and easy problems. In: IJCAI’95. (1995) 3 , 13 [46] Schrijv er, A.: A combinatorial algorithm minimizing submo dular functions in strongly p olynomial time. Journal of Combinatorial Theory , Series B 80 (2000) 346–355 2 [47] Schrijv er, A.: Com binatorial Optimization: Polyhedra and Efficiency . V olume 24 of Algorithms and Com binatorics. Springer-V erlag (2003) 1 , 2 , 5 [48] T opkis, D.: Sup ermo dularit y and Complementarit y . Princeton Univ ersity Press (1998) 1 [49] W ainwrigh t, M.J., Jordan, M.I.: Graphical mo dels, exp onen tial families, and v ariational inference. T echnical Rep ort 649, UC Berk eley , Dept. of Statistics (September 2003) 2 , 13 [50] W erner, T.: A Linear Programming Approac h to Max-Sum Problem: A Review. IEEE T ransactions on P attern Analysis and Machine Intelligence 29 (7) (2007) 1165–1179 13 [51] Zalesky , B.: Efficien t determination of Gibbs estimators with submo dular energy functions. arXiv:math/0304041v1 (F ebruary 2008) 2 , 8 [52] ˇ Zivn´ y, S., Jeav ons, P .G.: Classes of submodular constraints expressible b y graph cuts. In: Proceedings of the 14th International Conference on Principles and Practice of Con traint Programming (CP’08). V olume 5202 of LNCS. (2008) 112–127 2 , 8 [53] ˇ Zivn´ y, S., Jea vons, P .G.: Whic h submo dular functions are expressible using binary submo dular func- tions? Research Rep ort CS-RR-08-08, Computing Lab oratory , Universit y of Oxford, Oxford, UK (June 2008) 13 16