Exhaustive enumeration unveils clustering and freezing in random 3-SAT

Exhaustiv e en umeration un v eils clustering and freezing in random 3- SA T John Ardelius 1 and Lenk a Zdeborov´ a 2, 3 1 Swe dish Institute of Computer Scienc e SI CS, SE-164 24, Kista, Swe de n ∗ 2 Universit ´ e Paris-Sud, LPTMS, UMR8626, Bˆ at. 100, Universit´ e Paris-Sud 91405 Ors ay c e dex 3 CNRS, LPTMS, UMR8626, Bˆ at. 100, Uni versit ´ e Paris-Sud 91405 Orsay c e de x † (Dated: N ovem ber 2, 201 8) W e study geometrical prop erties of the complete set of s olutions of the random 3-satisfiabilit y problem. W e sho w that even for mo derate system sizes the num b er of clusters corresp onds surpris- ingly well with th e theoretic asymptotic prediction. W e locate th e freezing transition in the space of solutions which has b een conjectured t o b e relev ant in explaining the onset of computational hardness in random constraint satisfaction problems. P ACS num bers: 89.20.Ff,75.10.Nr,89.70.Eg Satisfiability (SA T) is one of the mos t impo rtant prob- lems in theoretical computer science. It was the first problem shown to b e NP-co mplete [1, 2], and it is of cen- tral rele v ance in v arious practica l a pplica tions, including artificial in telligence, planning, har dware and electronic design, automation, v erificatio n and mor e. It can thus be pictor ia lly thought of a s the Is ing mo del o f computer science. E nsembles of randomly ge nerated SA T instances emerged in computer sc ie nce a s a way o f ev aluating algo - rithmic p erfor mance and addressing questions reg a rding the av era ge case co mplexit y . An ins tance o f r andom K -SA T pro blem cons is ts of N Bo olean v ariables and M clauses. Each claus e contains a subset of K distinct v ariables chosen unifor mly at ran- dom, and e a ch clause for bids one ra ndo m ass ignment of the K v ariables out of the 2 K po ssible ones. The pro b- lem is satisfiable if there exists a v ariable a ssignment that simult aneo usly satisfies all clause s and we call suc h an assignments a solution to the problem. W hen the den- sity of constraints α = M / N is incr eased, the formulas bec ome less likely to be satisfiable. In the thermo dy- namical limit ther e is a sharp tra nsition from a phase in which the formulas are almost surely satisfia ble to a phase w her e they are a lmost sur ely unsa tisfiable. The ex- istence o f this transition is partly established rigorously [3]. It is also a well kno wn empirical res ult that the hard- est instanc e s are found near to this thres ho ld [4, 5, 6]. Random K -SA T ha s attracted interest of statistical ph ysicis ts bec a use o f its equiv alence to mean field spin glasses [7 ]. Indeed, the problem can b e rephra sed as minimizing a spin gla ss-like energy function which coun ts the num b e r o f violated clauses . The res ults and insights coming from this e q uiv alence are r emark able. The sa t- isfiability threshold a nd o ther phas e trans itions in the structure of solutions a re descr ib ed in [8 , 9, 10]. In par- ticular, it w as sho wn that for K ≥ 3 the space of solutions for highly constrained but still satisfiable instances splits int o exp o nentially ma n y clusters and in so me cas es this clustering has b een rigoro usly confirmed [11, 12]. The so-called freezing of v ariables in cluster s is another ric h concept studied recently [1 3, 14]. How ever, a detailed understanding of ho w the clustering o r freezing of solu- tions affects the av erage co mputational hardness is still one of the most int eres ting op en problems in the field. Since the exa ct statistical physics solution of the random sa tisfiability problem appea red [9, 15] dozens of directly rela ted articles followed. Mathematicians and co mputer sc ie ntist now adays regar d these analy tical works as a ric h source of results which a r e mos tly unac- cessible to the current proba bilistic methods. Y et none of these works tried to compare the ana lytical asymptotic predictions to numerical simulations o n a quantit ative level and numerical in vestigations mostly c oncentrated on per formance analysis of satisfiabilit y solvers. There- fore the relev ance of the asymptotic predictions for sys- tems of practical sizes, which in computer science are not at the scale o f the Avogadro num b er, remained a lmost un touched. Our letter aims at filling this g ap and to e n- couraging further inv estigation in this directio n. W e use conceptually relativ ely simple numerical tec hniques and yet obtain nontrivial results. W e present t w o of our mos t int eres ting findings. The first is a quantitative compar- ison b etw een the num b er o f clusters of solutions (g lassy states) a nd its analytical prediction [9, 1 5, 1 6, 17]. The second is the lo c ation o f the freezing transition which was recently sugg e sted to b e resp ons ible fo r computational hardness o f the random satisfiability problem [1 4, 18, 19], but not yet computed in the 3-SA T problem. Clustering and fr e ezing — In physics o f g lassy sys- tems, clus ters corres p ond to pure thermo dynamica l states and ar e b eing descr ibed in the literature ab out glasses and s pin glasses for more than one qua r ter of a centu ry [7]. A fo rmal de finitio n of clus ter s in K -SA T as extremal Gibbs mea sures was given r ecent ly in [10]. W e will re fer to these a s the c avity-clusters . It is not known, how ever, how to adapt th is definition to insta nc e s of fi- nite size. In this work, we define clusters a s connected comp onents in a graph where e ach solution is a vertex and whe r e e dg es connect solutio ns that differ in only one v ariable [37]. This definition is applica ble to a ny finite instance of the K -SA T problem. It is most likely not strictly equiv alent to the definition of the cavit y-clusters , 2 yet it r epro duces many of their prop erties. In order to shed light on the relation betw een ca vity- clusters and co nnected-comp onent clus ters we now in tro- duce the pro cedure called whitening a nd the conce pt of frozen v ariables. Whitening of a solution in K -SA T is defined in the following wa y [20]: star t with the s olution, assign iter atively a ” ∗ ” (joker) to v ariables which be lo ng only to clauses which ar e alr eady sa tisfied by another v ariable or a lr eady contain a ∗ v ar iable [38]. Whitening is in the liter ature referr ed also a s p eeling [2 1] or co ars- ening [12]. The fixed p oint o f this pro cedure is called a whitening-c or e , it is also referre d as cor e [12, 21], or true cov er [22]. A v ariable is said to b e fr ozen in a set of so - lutions if it takes only one v alue (either 0 or 1) in a ll the solutions in the set. Note that if the satisfia bilit y thresh- old is shar p there canno t b e a finite fr a ction o f v a riables frozen in all the so lutio ns in the satisfia ble reg ion [2 3]. On the other ha nd, v aria bles might be frozen in the indi- vidual clus ters. Acco rding to the cavity metho d [24, 25] this is indeed the ca s e and freezing of clusters hav e b een studied in [13, 1 4, 2 6]. According to the cavity metho d [2 4, 25] ther e is a deep connection b e t ween frozen v ar iables and the whitening- core: if the one-step r eplica symmetric solution is corr ect then on lar g e typical instances the s e t of fr ozen v a riables in the cavit y-cluster and the non- ∗ part o f the whitening core a re ide ntical [9 ]. Thus the whitening cor es o f all so- lutions belonging to one ca vity-cluster a re identical. This also ho lds for the co nnec ted-comp onent clusters: Indeed, t wo solutions that differ in a single v aria ble ha ve the same whitening core since the whitening can b e started from that sp ecific v ariable [39]. F urther, v aria ble s b elonging to the whitening cor e m ust b e frozen in the connected- comp onent cluster, the opp osite implication is in genera l not true [40]. Two additiona l rema rks ab out clus ters are imp orta nt. First, whitening cor es are sometimes wrongly identified with c lusters. In part o f the clustered phase almos t all so - lutions b elong to soft (unfrozen) cavit y-cluster s [14, 26]. In par ticular in 3- SA T this seems to b e the case a t least up to constra in t density α = 4 . 25 [2 7]. Second, it seems that a ll k nown heuris tic algo rithms need an ex po nen- tial time to find solutions with a non-trivia l (not all- ∗ ) whitening cor es, see e.g. [14, 21, 28]. This motiv ates our study of the fr e ezing tr ansition , α f . It is defined as the smallest densit y of constra in ts α suc h that all solu- tions b elong to fr o zen clusters, i.e., their whitening core is not made fr om all- ∗ . W e use the whitening co r e in- stead of the r eal s et of frozen v ar iables, b ecause in sma ll instances there are almost a lwa y s at least few fro zen v a ri- ables. Existence of the fro zen phase was prov en in the thermo dynamical limit for K ≥ 9 near to the s atisfiabil- it y thr e shold in [12]. Sev era l theoretical inv estiga tio ns of a related rigidit y transition, where clusters which contain almost all the solution b ecome frozen, can b e found in [13, 14, 2 6]. But a s lo ng as soft clusters exist so me algo- rithms may b e a ble to find them, as shown in [19]. A re- lated numerical study [2 2] inv estigates the size de p en- dence of the fraction of froz e n solutions at α = 4 . 2 0 < α f . The c omplexity function — W e ge ne r ate instances of random 3-SA T problems with N v ar iables and M clauses using the m akewf f program [29]. The num b er of solutions is then calcula ted using the exha ustive search metho d relsat [30] and the complete s et of solutions is clus ter ed through br e ath first se ar ch . This works as follows: W e o r- der the N s olutions in binary lexical order . F urther, for all the s olutions we genera te all the N neighboring config - urations, sear ch them in the list a nd if found concatenate the t wo in the s ame cluster re s ulting in an a lgorithmic complexity of O ( N log 2 N ), considering that log N ≈ N . 0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.4 complexity density of constraints N=25 N=50 N=75 N=100 N=125 N=150 SP 0 0.004 0.008 4.2 4.25 4.3 4.35 FIG. 1: The av erage complexity function, logarithm of the num b er of connected-comp onent clusters divided by N , for different system sizes compared to th e asymptotic prediction [15, 17 ]. Note that th e numerical curves will conti nue to muc h lo wer v alues of α than plotted. In order to obtain information ab out cluster s in a t yp- ical formula w ith N v ar iables and M clauses , we co un t the num b e r of solutions in A = 999 s uch random fo r mu - las and select the media n instance in terms of num b er of so lutions on which we then count the num b e r o f clus- ters S . This is rep eated B = 100 0 times. The complexity function Σ( N ) = h lo g S i / N is then co mputed a s average of the logarithm of the n umber of clusters divided by system size N . If the median instance is unsa tisfiable it c o nt ributes a zer o v alue to the av e rage, this do es not hav e influence of the asymptotic v alue. T aking the me- dian has tw o impo rtant adv antages, fir st we a void rare formulas with very many so lutions which are n umer ically int ra c ta ble, se c o nd the complexity converges v ery fast to zero in the unsatisfia ble reg ion. The result is plotted in Fig. 1 and co mpared with the asymptotic co mplexit y function computed from the survey propaga tion equa- tions, which in 3-SA T gives a non-zero r esult for α > 3 . 9 2 [15, 17]. The agr eement is remark a bly g o od, in particular around the s a tisfiability thresho ld α s = 4 . 267 [9, 17]. 3 It was discussed in [10], and shown n umerica lly in [22], that clusters exist even fo r α < 3 . 92. W e indeed do no t see anything particular happ ening a t α = 3 . 92 . Below the c lustering tr ansition, α < 3 . 86, ho wev er, the large s t cavit y-cluster should con tain almost all the solutions [1 0]. W e see a corresp onding tr end in the a verage fra ction of solutions cov ered by the la rgest cluster in our data. It should also b e mentioned that the s urvey pr opagatio n prediction is b elieved to b e ex a ct only for α > 4 . 15 [16]. The fr e ezing tr ansition — In order to determine the freezing transition we start with a formula of N v ariables and all p ossible clauses, a nd remov e the clauses one b y one independently at random. W e mar k the n umber of clauses M s where the formula b e comes satisfia ble a s well as the num b er o f clauses M f ≤ M s where at least one so - lution starts to hav e an all- ∗ whitening core. W e repe a t B -times ( B = 2 · 10 4 in Fig. 2) and compute the pro babil- ities that a formula of M clauses is satisfiable P s ( α, N ), and unfrozen P f ( α, N ), resp ectively . Due to memory lim- itation we can treat only instances which have less than 5 · 10 7 solutions which limits us to s ystem sizes N ≤ 1 00. Our results for the satis fia bilit y threshold are consis ten t with previo us studies in [6, 23, 3 1]. The proba bility of being unfro zen, P f ( α, N ), is shown in Fig. 2. It is tempting to perfor m a scaling analysis as has bee n done in [6, 23, 3 1] for the satisfia bilit y threshold. The critical exp onent related to the width of the scal- ing window was defined via resca ling of v ariable α as N 1 /ν s (1 − α/ α s ( K, N )). Note, howev er , that the esti- mate ν s = 1 . 5 ± 0 . 1 for 3- SA T provided in [3 1] is not the cor rect a symptotic v alue. It was prov en in [32] that ν s ≥ 2. Indeed it was sho wn n umerically in [33] that a crossover exis ts at sizes o f o rder N ≈ 10 4 in the related X OR-SA T pro blem. A similar situation happ ens for the scaling of the freezing trans ition, P f ( α, N ), as the pro of of [32] applies also here [4 1]. It would be in teresting to in- vestigate the scaling behavior on an ensem ble of insta nces where r esults o f [3 2] do not apply . Here w e concen trate instead o n the estimation of the cr itical p oint, which we presume not to b e influenced by the crossover in the sca l- ing. W e ar e in a muc h mor e conv enient situation than for the satisfiability transitio n. The crossing point for the functions P f ( α, N ) of different system sizes seems not to depe nd on N , while for the satisfiability transition its size dep endence is very strong [3 1]. W e determine the v a lue o f the freezing transition as α f = 4 . 25 4 ± 0 . 0 09, which is extremely clo se to the s at- isfiability thre s hold α s = 4 . 267 [15]. Analytical study suggests α f > 4 . 25 [27]. W e exp ect the t wo tra nsitions to b e separ ated α f < α s [12, 14, 26], and Fig . 2 s uggests so but it is o n the b order of statistica l sig nificance. How- ever, the main motiv ation to study the freezing transition is its p otential connection to the onset o f algorithmical hardness [14, 18, 19]. W e thus co mpare its v alue with the estimates o f p erforma nce of the b est a lgorithms known for random 3 -SA T. The leading sto chastic lo cal search 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 3 3.5 4 4.5 5 5.5 probability unfrozen density of constraints α s α d N=25 N=35 N=45 N=55 N=65 N=80 N=100 0.24 0.26 0.28 0.3 0.32 0.34 0.36 0.38 4.2 4.22 4.24 4.26 4.28 4.3 4.32 probability unfrozen density of constraints α s α f SP SLS N=25 N=35 N=45 N=55 N=65 N=80 N=100 FIG. 2: T op: Probabilit y t h at there ex ists an unfrozen so- lution as a function of th e constraint density α f or differen t system sizes. The clustering [10] and satisfiability [9] transi- tions are mark ed for comparison. Bottom: A 1:20 zo om on the critical (crossing) point, our estimate for the freezing tran- sition is α f = 4 . 254 ± 0 . 009. The curves are cubic fits in the interv al (4 , 4 . 4). The arro ws represen t estimates of the li m- its of p erformance of t h e b est known sto chastic lo cal sea rch [28, 34] and survey propagation [35 , 36] algorithms. algorithms work in linea r time up to α = 4 . 21 [28, 34]. The surv ey pro pagation (SP) decimation was es timated to work up to α = 4 . 252 [3 5], the same point was de- termined as the limit of the SP reinforc emen t [36]. The agreement b etw een o ur lo cation of the freez ing transi- tion and the p erfor mance of SP supp orts stro ngly the conjecture that the frozen phase is har d for an y known algorithm. In random 3-SA T this regio n is very nar r ow, in contrast to the s itua tion in K ≥ 9 SA T [1 2]. Discussion — The ma in contribution of this work is the demonstr a tion that the asymptotic predictio ns com- ing fr om the statistica l physics analy s is are relev a nt even for instances of very mo der ate size. In particular , w e pre- sented a numerical compar ison b etw ee n the nu mber of connected-comp onent clusters and the a symptotic pre- diction for the co mplexit y function in random 3-SA T and o bta in a r emark a bly go o d agreement. F urthermor e , 4 we estimate the loca tion of the freezing transition a t α f = 4 . 254, which is consis ten t w ith the p erfor mance threshold of the b est known algor ithms. W e also s how that exhaustive en umeratio n, despite its curr en t size lim- itations, is a p ow erful to ol to study r andom optimiza tio n problems: indeed the kno wledge of the complete set of so- lutions a llows to tac kle questions that ar e complemen tary to those answered by c lassical Monte-Carlo metho ds. The definitions of clusters a nd the whitening co re, that we a do pted, is applicable to any instance of the s a tisfia- bilit y problem. As such, they offer an interesting dir e c- tion for future resea rch of r eal-world K -SA T instances . In addition, we observe that the properties related to clus- tering ar e less sensitive to finite-s ize effects than the one s related to the solutions themselves. This is int eres ting and cer tainly worth further in vestigations. F uture w ork could als o cover 2-SA T, wher e the solutions are muc h more numerous ev en for v ery sma ll sys tem s izes, or K - SA T with K > 3, where larger form ulas will b e needed to inv estiga te the relev a nt reg ions, how ever, the freez ing transition is more separa ted from the satisfia bilit y when K g r ows. The n umerica l loca tion of the clustering and condensation tr ansitions [10] is also o f interest. A cknow le dgment — W e tha nk Stephan Mertens for sharing the data from [1 7]. W e also gratefully acknowl- edge Thomas J o erg, Flo rent Krzak ala, Marc M´ ezar d and F ederico Ricc i-T erseng hi for ma n y prec io us discussions and comments. This work was partia lly supp or ted by FP6 prog ram EVERGR OW. J.A. thanks KITP C-CAS for hospitality . ∗ Electronic address: john@sics.se † Electronic address: zdeb orov@lptms.u-psud.fr [1] S. A. Cook, in Pr o c. 3r d STOC (ACM, New Y ork, NY , USA, 1971), pp. 151–158. [2] C. H. Papadimitriou, Computational c omplexity (Addison-W esley , 1994). [3] E. F riedgut, J. A mer. Math. S oc. 12 (1999). [4] P . Cheeseman, B. Kanefsky , and W. M. T aylor, in Pr o c. 12th IJCAI (Morgan K aufmann, San Mateo, CA, USA , 1991), pp. 331–337. [5] D. G. Mitchell, B. Selman, and H. J. Levesque, in Pr o c. 10th AAAI (A A AI Press, Menlo Park, Cali fornia, 1992), pp. 459–465. [6] S. Kirkpatrick and B. Selman, Science 264 , 1297 (1994). [7] M. M´ ezard, G. Parisi , and M. A. V irasoro, Spin-Glass The ory and Beyond , vol. 9 of L e ctur e Notes in Physics (W orld Scientific, S ingapore, 1987). [8] G. Biroli, R. Monasson, and M. W eigt, Eur. Phys. J. B 14 , 551 (2000). [9] M. M´ ezard, G. Pari si, and R. Zecchina, Science 297 , 812 (2002). [10] F. Krzaka la, A. Mon tanari, F. R icci-T ersenghi, G. Se- merjian, and L. Zd eborov´ a, Pro c. Natl. A cad. Sci. 104 , 10318 (2007). [11] M. M´ ezard, T. Mora, and R. Zecchina, Phys. Rev. Lett. 94 , 197205 (2005). [12] D. Achlioptas and F. Ricci-T ersenghi, in Pr o c. of 38th STOC (ACM, New Y ork, N Y, USA, 2006), pp . 130–139 . [13] G. S emerjian, J. S tat. Phys. 130 , 251 ( 2008). [14] L. Zdeb orov´ a and F. Krzak ala, Phys. R ev. E 76 , 031131 (2007). [15] M. M´ ezard and R. Zecchina, Phys. Rev. E 66 , 05612 6 (2002). [16] A. Mon t anari, G. Parisi, and F. R icci-T ersenghi, J. Ph ys. A 37 , 2073 (2004). [17] S. Mertens, M. M´ ezard, and R. Zecchina, R and. S t ruct. Algo. 28 , 340 (2006). [18] F. Krzak ala and J. Kurchan, Phys. R ev . E 76 , 021122 (2007). [19] L. D all’Asta, A. Ramezanp our, and R . Zecchina, Ph ys. Rev. E 77 (2008). [20] G. Parisi (2002), arXiv:cs.CC/0 212047. [21] E. N. Manev a, E. Mossel, and M. J. W ainwrigh t, J. ACM 54 (2007). [22] L. Kro c, A. S abharw al, and B. Selman, in Pr o c. of 23r d AUAI ( AUAI Press, A rlington, Virginia, USA , 2007), pp. 217–226. [23] R. Monasson, R. Zecc hina, S. Kirk patric k, B. S elman, and L. T ro yansky , Nature 400 , 133 (1999). [24] M. M´ ezard and G. Parisi, J. Stat. Phys. 111 , 1 (2003). [25] M. M´ ezard and G. P arisi, Eur. Phys. J. B 20 , 217 (2001). [26] A. Montanari, F. R icci-T ersenghi, and G. S emerjian, J. Stat. Mech. p. P04004 (2008). [27] F. Ricci-T ersenghi, priv ate communication. [28] S. Seitz, M. Alav a, and P . Orp onen, J. Stat. Mech. p. P06006 (2005). [29] F oun d at: http://www.cs .ro chester. edu/ k au t z/w alksat/. [30] R. J. Bay ardo and J. D. Pehousek, in Pr o c. 17th AAAI (AAAI Press, Menlo Park, California, 2000), pp. 157– 162. [31] R. Monasson, R. Zecc hina, S. Kirk patric k, B. S elman, and L. T ro yansky , Rand . Struct . Algo. 15 , 414 ( 1999). [32] D. B. Wilson, R and. Struct. Algo. 21 , 182 (2002). [33] M. Leone, F. R icci-T ersenghi, and R. Zecchina, J. Phys. A 34 , 4615 (2001). [34] J. Ardelius and E. Au rell, Phys. Rev . E 74 , 037702 (2006). [35] G. Parisi (2003), arXiv:cs/03010 15. [36] J. Chav as, C. F urtlehn er, M. M ´ ezard, and R. Zecc h in a, J. S t at. Mech. p. P11016 (2005). [37] Another common choice of d istance b etw een neighbor- hoo ding solutions is ”any sub-extensive distance”. This choi ce is, how ever, problematic for fi nite-size in stances and lac ks the important prop erty t h at all the s olutions in a cluster hav e the same whitening core. In some other mod els this defin ition is to b e mo dified, e.g., in th e 1-in- K SA T the minimal distance b etw een solutions is 2. [38] In a general constraint satisfaction p roblem t h e whiten- ing mush b e defi ned via the warning propagation. [39] W e hav e not fo und any general argument w hy t wo dif- feren t connected- comp on ent clusters could n ot hav e the same non-all- ∗ whitening core, b u t w e hav e not observed any such case in our data. [40] W e are aw are of one property which the ca v it y-clu sters migh t ha ve bu t which cannot b e reproduced asymptot- ically with the connected- compon ent clusters, namel y a purely entropic separation. [41] Theorem 1 of [32] app lies to t h e freezing prop erty where the bystander are clauses containing tw o leav es.