Algorithmically independent sequences

Algorith mic all y Indep enden t Sequence s Cristian S. Calude ∗ Marius Zimand † Abstract Two ob jects ar e indep endent if they do not affect each other. Independenc e is w ell- understo o d in classical infor mation theor y , but less in a lgorithmic information theory . W orking in the fr amework o f algorithmic informatio n theory , the pap er pro po ses tw o t yp e s of indep endence for a rbitrary infinite binary sequence s and studies their prop er- ties. Our tw o prop os ed no tions of indep endence ha ve some of the in tuitiv e proper ties that one naturally e x pec ts . F or example, for every sequence x , the set o f sequences that are indep endent (in the weak er of the tw o senses) with x has measure one. F or bo th notions of indepe ndence w e inv estigate to wha t extent pairs of independent se- quences, ca n b e effectively cons tr ucted via T uring reductions (from one or more input sequences). In this resp ect, we prove several imp ossibility results. F or example, it is shown that there is no effective wa y of pro ducing fro m an a rbitrary seq uence with po sitive co nstructive Hausdorff dimension tw o sequences that are indep endent (even in the weak er t ype of indep endence) and have super- logarithmic complexity . Finally , a few conjectures and op en q uestions a re discus s ed. 1 In tro d uction In tuitiv ely , t w o ob jects are ind ep endent if they do not affect eac h other. The concept is w ell-understo o d in classic al in formation theory . There, the ob jects are random v ariables, the information in a random v ariable is its S hannon en t rop y , and t wo random v ariables X and Y are declared to b e indep end en t if the information in the join ( X, Y ) is equal to th e sum o f t he informatio n in X and the inf ormation in Y . This is equiv al en t to sa ying that the information in X conditioned by Y is equal to the inform ation in X , with th e interpretation that, on a v erage, kno wing a particular v alue of Y do es not affect the inf ormation in X . The notion of indep end ence h as b een defined in algorithmic information theory as w e ll for fin ite strings [Cha82]. The app roac h is very similar. This time the information in a string x is the complexit y (plain or prefix-fr ee) of x , and t w o strings x and y are indep enden t ∗ Department of Computer Science, Universit y of Auckland, New Zealand, www.cs.auckla nd.ac.nz/ ~cristian . Calude was supp orted in part by UARC Grant 3607894/ 9343 and CS-PBRF Grant. † Department of Co mputer and Information S ciences, T owso n Universit y , Baltimore, MD, US A , http://tri ton.towso n.edu/ ∼ mzimand . Zimand was sup p orted by NSF grant CCF 06348 30. Part of th is w ork was done while v isiting the CDMTCS of the Universit y of Auckland, New Zealand. 1 if the information in the join s tring h x, y i is equal to the s um of the information in x and the inform ation in y , up to logarithmic (or, in some cases, constan t) precision. The case of in finite sequences (in short, s equences) has b een less studied. An insp ection of the l iterature rev ea ls that for this s etting, indep endence has b een co nsidered to b e synon ymous with pairwise relativ e rand omness, i.e., t w o sequences x and y are said to b e indep end en t if they are (Martin-L¨ of ) rand om r elativ e to ea c h other (see [vL90, DH ]). The effect of this approac h is that the notion of ind ep endence is co nfined to the situation where the sequences are random. The main ob j ectiv e of th is pap er is to p u t forward a concept of ind ep endence th at applies to al l sequences. One can en vision v arious w a ys for d oing this. One p ossibilit y is to use Levin’s notion of mutual information for sequences [Lev84] (see also the surv ey pap er [GV04]) and declare tw o s equences to b e indep endent if their mutual information is small. If one purs u es this direction, the main issue is to determine the righ t definition for “small.” W e tak e another ap p roac h, w hic h consists in extending in the natural wa y the notion of indep endence fr om fin ite strings to sequences. This leads us to t wo con- cepts: indep endenc e and finitary-indep endenc e . W e say that (1) t wo s equ ences x and y are in dep endent if, for all n , the complexit y of x ↾ n (the prefix of x of length n ) and the complexit y of x ↾ n relativized with y are within O (log n ) (and the same relation h olds if w e swap the roles of x and y ), and (2) tw o sequences x and y are fi nitary-indep end en t if, for all n and m , the complexit y of x ↾ n and the complexit y of x ↾ n giv en y ↾ m are within O (log n + log m ) (and the same relation holds if we sw ap the roles of x and y ). W e hav e settled for the additiv e logarithmical term of precision (rather than some higher accuracy) since this p ro vides r obustness with resp ect to the type of complexit y (plain or p refix-free) and other tec h nical adv anta ges. W e establish a series of b asic facts r egardin g the prop osed n otions of indep en dence. W e sho w that indep endence is strictly stronger than fin itary-indep endence. The tw o n otions of indep end ence app ly to a larger category of sequences than the family of random sequences, as inte nded. Ho wev er, they are to o rough for b eing relev ant for computable sequences. It is not hard to see that a computable sequence x is indep enden t w ith any other sequence y , simply b ecause the inform ation in x can b e obtained d ir ectly . In fact, this type of trivial indep end ence holds for a larger t yp e of sequences, namely for an y H -trivial s equence, and trivial finitary-indep endence holds for an y s equence x whose p r efixes h av e logarithmic com- plexit y . It seems that for this t yp e of sequences (computable or with v ery lo w complexit y) a more refined defin ition of in dep enden ce is needed (p erh aps, based on resource-b oun ded complexit y). W e sho w that the t w o pr op osed n otions of in dep endence ha ve some of th e in tuitive prop erties that one naturally exp ects. F or examp le, for ev ery sequence x , the set of sequences that are fin itary-indep endent with x has measure one. The same iss u e f or indep end ence remains op en. W e n ext in v estigate to what extent p airs of indep end ent, or finitary-indep end en t se- quences, can b e effectiv ely constructed via T urin g reductions. F or e x amp le, is there a T u ring r ed uction f that giv en orac le access to an arbitrary sequence x p ro duces a s equ ence 2 that is finitary-ind ep endent with x ? Clearly , if we allo w the output of f to b e a computable sequence, then the answer is p ositiv e by the typ e of trivial finitary-indep en dence that we ha ve noted ab o v e. W e sho w that if w e insist that the output of f has sup er-logarithmic complexit y w henev er x has p ositiv e constructiv e Hausdorff dimen s ion, th en the answ er is negativ e. In the same v ein, it is sho wn th at there is n o effect iv e w a y of pro ducing from an arbitrary sequence x with p ositiv e constructiv e Hausdorff dimen sion t w o s equences th at are fin itary-ind ep endent and ha ve sup er-logarithmic complexit y . Similar questions are considered f or the situation when w e are giv en t wo (finitary-) indep end en t sequences. It is sho wn that there are indep end ent sequences x and y and a T u ring r eduction g such that x and g ( y ) are not in dep endent. This app ears to b e a bad artifact of th e notion of indep endence prop osed in this pap er. W e consider that this is the only coun ter-in tuitiv e effect of our d efinitions. W e do not kno w if a similar phenomenon holds for finitary-indep end ence. On the other hand , for any indep enden t sequen ces x and y and f or an y T uring reduction g , x and g ( y ) are fi n itary-indep endent. W e also raise the question on w hether give n as input sev eral (fin itary-) indep enden t sequences x and y it is p ossible t o effectiv ely bu ild a new sequence that is (finitary-) indep end en t (not in the trivial w ay) with eac h sequence in the inpu t. It is observe d that the answer is p ositiv e if the sequ ences in the input are ran d om, but for other t y p es of sequences the question remains op en. The same iss u e can b e raised regarding finite strin gs and for this ca se a p ositiv e answ er is o b tained. Namely , it is s h o wn that giv en thr ee indep end en t finite strin gs x , y and z with linear complexit y , one can effectiv ely construct a new strin g that is indep enden t with eac h of x, y and z , and has high complexit y and length a constan t fraction of the length of x, y and z . 1.1 Preliminaries N denotes the se t of non-negativ e int egers; th e size of a finite set A is denoted || A || . Unless stated otherwise, all n u m b ers are in N and all logs are in b ase 2. W e wo r k o ve r the binary alphab et { 0 , 1 } . A strin g is an elemen t of { 0 , 1 } ∗ and a sequence is an element of { 0 , 1 } ∞ . If x is a string, | x | denotes its length; xy denotes the concate- nation of the strin gs x and y . If x is a strin g or a sequence, x ( i ) denotes the i -th bit of x and x ↾ n is the substring x (1) x (2) · · · x ( n ). F or t wo sequences x and y , x ⊕ y denotes the sequence x (1) y (1) x (2) y (2) x (3) y (3) · · · and x X O R y d en otes the sequence ( x (1) XOR y (1))( x (2 ) X OR y (2))( x (3 ) X OR y (3)) · · · , where ( x ( i ) X OR y ( i )) is th e sum mo dulo 2 of the bits x ( i ) and y ( i ). W e iden tify a sequence x with the set { n ∈ N | x ( n ) = 1 } . W e say that a s equ ence x is computable (computably en umer ab le, or c.e.) i f the corre- sp ondin g set is computable (resp ectiv ely , computably enumerable, or c.e.). If x is c.e., then for ev ery s ∈ N , x s is the sequ ence corresp onding to the set of elements en um er ated within s steps b y some mac hine M that en umerates x (the mac hine M is give n in the con text). W e also identify a sequence x with the real n u m b er in the in terv al [0 , 1] whose binary writing is 0 .x (1 ) x (2) · · · . A sequence x is s aid to b e left c.e. if the corresp ondin g 3 real num b er x is the limit of a computable incr easing sequ en ce of rational num b er s . Th e plain and the prefix-free complexities of a string are d efi ned in the standard w a y; how ev er w e need to pr o vide a few d etails regarding the compu tational mo dels. The mac hines that w e consider pro cess information giv en in three forms: (1) the input, (2) the oracle set, (3) the conditional str ing. Corresp ondingly , a unive rsal machine has 3 tap es: • one tap e f or the inpu t and w ork, • one tap e f or storing the cond itional string, • one tap e (called the oracle-query tap e) for formulating queries to the oracle. The oracle is a string or a sequence. If th e mac hine en ters the query s tate an d the v alue written in binary on the oracle-query tap e is n , then the mac hine gets the n -th bit in the oracle, or if n is larger th an the length of the oracle, the mac hine en ters an infin ite loop. W e fix su c h a universal mac hin e U . T he notatio n U w ( u | v ) m eans that th e input is u , the conditional str in g v and the oracle is giv en b y w , which is a string or a s equence. The plain c omplexit y of a string x giv en the oracle w and the conditional string v is C w ( x | v ) = min {| u | | U w ( u | v ) = x } . There exists a constan t c such that for ev ery x, v and w C w ( x | v ) < | x | + c . A mac hine is prefix-free (self-delimiting) if its domain is a prefix-free set. There exist unive r sal prefix-fr ee mac h ines; we fi x suc h a m achine U ; the prefi x -fr ee complexit y of a string x giv en the oracle w and the conditional string v is H w ( x | v ) = m in {| u | | U w ( u | v ) = x } . In case w or v are the empt y strin gs, w e omit them in C ( · ) and H ( · ). Throughout this pap er w e use the O ( · ) notation t o hid e constan ts t hat d ep end only on the c hoice of th e u niv ersal mac hine underlying the definitions of the complexities C and H . Since the prefix-free universal mac hin e is a particular typ e of mac hine, it follo ws that C w ( x | v ) < H w ( x | v ) + O (1), for every x, v and w . The r ev erse in equalit y b et w een C ( · ) and H ( · ) also h olds true, within an additiv e logarithmic term, and is obtained as follo ws. F or examp le, a string x = x (1) x (2) · · · x ( n ) can b e co ded in a self-delimiting w ay b y x 7→ code ( x ) = 11 · · · 1 | {z } | bin ( n ) | 0bin( n ) x (1) x (2) · · · x ( n ) , w here bin( n ) is the binary represent ation of n ∈ N . Note that | cod e ( x ) | = | x | + 2 log | x | + O (1). T his implies that for every x, v , and w , C w ( x | v ) > H w ( x | v ) − 2 log | x | − O (1) . (1) The follo wing inequalities h old for all strin gs x and y : C y ( x ) ≤ C ( x | y ) + 2 log | y | + O (1) , (2) | C ( xy ) − ( C ( x | y ) + C ( y )) | ≤ O (log C ( x ) + log C ( y )) . (3) 4 The fi rst inequ alit y is easy to der ive directly; the second one is called the Symm etry of Information Theorem, see [ZL70]. There are v arious equiv alen t defin itions f or (algorithmic) r andom sequences as defined b y Martin-L¨ of [ML66] (see [C 02]). In what follo ws we w ill use the (we ak) complexit y- theoretic one [Cha75] using the p r efix-free complexit y: A sequence x is Martin-L¨ of random (in sh ort, random) if there is a constan t c suc h that for every n , H ( x ↾ n ) ≥ n − c . The set of rand om sequences has constructive (Leb esgue) measur e one [ML66]. The s equ ence x is random relativ e to the sequence y if there is a constant c su ch that for ev ery n , H y ( x ↾ n ) ≥ n − c . Note that if x is random, then for ev ery n , C ( x ↾ n ) ≥ n − 2 log n − O (1) (b y inequalit y (1)). A similar in equ alit y also holds f or the relativize d complexities, i.e. for all x that are rand om relativ e to y and for all n , C y ( x ↾ n ) > n − 2 log n − O (1). These results w ill b e rep eate dly used throughout the p ap er. In [vL90] v an Lambalge n prov es that x ⊕ y is random iff x is random and y is random relativ e to x . This implies that if x is random and y i s random relativ e to x then x is random relativ e to y . The c onstructiv e Hausd orff dimension of a sequence x —which is the di- rect effect ivization of “classical Hausdorff dimension”—d efi ned b y dim( x ) = lim inf n →∞ C ( x ↾ n ) /n (= lim inf n →∞ H ( x ↾ n ) /n ), measures intermediate lev els of rand om- ness (see [Rya8 4 , Sta93, T ad02, Ma y02, Lut03, Rei04, Sta05, CST06, DHNT06]). A T urin g red u ction f is an orac le T uring machine; f ( x ) is the language compu ted by f with oracle x , assumin g that f halts on all inputs when working with oracle x (otherwise w e say that f ( x ) do es not exist). In other words, if n ∈ f ( x ) then the mac hine f on in put n and with oracle x halts and outputs 1 and if n 6∈ f ( x ) then the mac hin e f on input n and w ith oracle x halts and ou tp uts 0. The f u nction use is defin ed as follo ws: us e x f ( n ) is the index of the righ tmost p osition on th e tap e of f acce ssed d uring the computation of f w ith oracle x on inpu t n . The T uring reduction f is a wtt-r e duction if there is a computable function q suc h that use x f ( n ) ≤ q ( n ), f or all n . The T uring reduction f is a truth-table r e duction if f halts on all in p uts f or eve r y oracle. A tru th-table r eduction is a wtt-reduction. 2 Defining indep endence The basic idea is to declare that t wo ob j ects are ind ep endent if none of them con tains significan t information ab out the other one. Thus, if in some formalization, I ( x ) d enotes the information in x and I ( x | y ) denotes t he information in x giv en y , x and y are indep end en t if I ( x ) − I ( x | y ) and I ( y ) − I ( y | x ) are b oth s m all. In this pap er we w ork in the framew ork of algo rithmic information theory . In this s etting, in case x is a string, I ( x ) is the complexit y of x (wh ere f or the “c omp lexit y of x ” there are seve ral p ossibilities, the main ones b eing the p lain complexit y or the p r efix-free complexit y). The indep endence of strings w as studied in [C ha82]: t wo strings are indep end en t if 5 I ( xy ) ≈ I ( x ) + I ( y ). Th is approac h motiv ates our Definition 2.1 and Definition 2.2. In case x is an infin ite sequence, the in formation in x is c h aracterized by the sequence ( I ( x ↾ n )) n ∈ N of information in the initial segmen ts of x . In the infin ite case, for the infor- mation up on whic h w e condition (e.g., the y in I ( x | y )), there are tw o p ossibilities: either the en tire sequence is a v ailable in th e form of an oracle, or w e consider initial segmen ts of it. Acco rdingly , we prop ose tw o notions of indep en dence. Definition 2.1 ( The “integral” type of indep endence ) Two se quenc es x and y ar e indep end en t if C x ( y ↾ n ) ≥ C ( y ↾ n ) − O (log n ) and C y ( x ↾ n ) ≥ C ( x ↾ n ) − O (log n ) . Definition 2.2 ( The finitary type of indep endence ) Two se quenc es x, y ar e finitary- indep end en t if for al l natur al numb ers n and m , C ( x ↾ n y ↾ m ) ≥ C ( x ↾ n ) + C ( y ↾ m ) − O (log ( n ) + log ( m )) . Remark 1 W e will sho w in Prop ositio n 2.4, that the inequalit y in Definition 2. 2 is equ iv- alen t to sa ying that for all n and m , C ( x ↾ n | y ↾ m ) ≥ C ( x ↾ n ) − O (log n + log m ), whic h is the fin ite analogue of the prop ert y in Definition 2.1 and is in line w ith our discuss ion ab o ve. Remark 2 If x and y are indep end ent, then they are also fin itary-ind ep endent (Prop osi- tion 2.5). The conv erse is not true (Corollary 4.13). Remark 3 The p rop osed d efinitions u se the plain complexit y C ( · ), but we could ha ve used the prefix-free complexit y as well, b ecause the t wo t yp es of complexit y are within an add itive logarithmic term. Also, in Definition 2.2 (and throughout this pap er ), w e use concatenati on to represent the joining of t wo strings. Ho wev er, since any reasonable p airing function h x, y i satisfies | |h x, y i| − | xy | | < O (log | x | + log | y | ), it follo ws that | C ( < x, y > ) − C ( xy ) | < O (log | x | + log | y | ), and th u s an y reaso nable pairing function could ha ve b een used instead. Remark 4 A deb atable issu e is the subtraction of the logarithmic term. Indeed, there are other natural p ossibilities. W e argue that our c hoice has certain adv antag es o ve r other p ossibilities that come to mind. Let us fo cus on the defin ition of fin itary-ind ep endence. W e w an t C ( x ↾ n y ↾ m ) ≥ C ( x ↾ n ) + C ( y ↾ n ) − O ( f ( x ) + f ( y )), for all n, m , where f shou ld b e some “small” fun ction. W e w ou ld lik e th e follo wing t wo pr op erties to hold: (A) the sequences x and y are fi nitary-indep end en t iff C ( x ↾ n | y ↾ m ) > C ( x ↾ n ) − O ( f ( x ↾ n ) + f ( y ↾ m )), for all n and m , (B) if x is “somewhat” random and y = 0 ω , then x and y are finitary-indep end en t. 6 Other natural p ossibilities for the definition could b e: (i) if f ( x ) = C ( | x | ), th e defin ition of finitary indep endence–(i) w ould no w b e: C ( x ↾ n y ↾ m ) ≥ C ( x ↾ n ) + C ( y ↾ m ) − O ( C ( n ) + C ( m )) , or (ii) if f ( x ) = log C ( x ), the defin ition of finitary-ind ep endence–(ii) w ould no w b e: C ( x ↾ n y ↾ m ) ≥ C ( x ↾ n ) + C ( y ↾ m ) − O (log C ( x ↾ n ) + log C ( y ↾ m )) . If sequences x and y satisfy (i), or (ii), then they also s atisfy Definition 2.2. V ariant (i) implies (B) , bu t n ot(A) (for example, consid er sequ en ces x and y with C ( n ) << log C ( x ↾ n ) and C ( m ) << log C ( y ↾ m ), for infin itely many n and m ). V ariant (ii) implies (A), but do es not imply (B) (for example if for infinitely man y n , C ( x ↾ n ) = O (log 3 n ); tak e such a v alue n , let p b e a shortest description of x ↾ n , and let m b e the in teger whose b inary r ep resen tation is 1 p . Then x ↾ n and 0 ω ↾ m , do not satisfy (B)). The prop osed defin ition implies b oth (A) and (B). Another adv an tage is the robustness prop erties fr om Remark 3. Remark 5 If th e sequen ce x is computable, then x is ind ep endent with ev ery sequence y . In fact a stronger fact holds. A sequence is called H -trivial if, for all n , H ( x ↾ n ) ≤ H ( n ) + O (1). T his is a notion that has b een in tens ively studied recen tly (see [DHNT06]). C learly ev ery computable sequence is H -trivial, but the conv erse do es not hold [Z am90, Sol75]. If x is H -trivial, then it is ind ep endent with ev ery sequence y . Indeed, H y ( x ↾ n ) ≥ H ( x ↾ n ) − O (log n ), b ecause H ( x ↾ n ) ≤ H ( n ) + O (1) ≤ log n + O (1), and H x ( y ↾ n ) ≥ H ( y ↾ n ) − O (log n ), b ecause, in fact, H x ( y ↾ n ) and H ( y ↾ n ) are within a constant of eac h other [Nie05]. The same inequalities hold if we use the C ( · ) complexit y (see Remark 3). F or the case of fi nitary-indep end en ce, a similar phenomenon holds for a (seemingly) ev en larger class. Definition 2.3 A se quenc e x is c al le d C-lo garithmic if C ( x ↾ n ) = O (log n ) . It can b e shown (for example using Prop osition 2. 4, (a)) that if x is C-logarithmic, then it is finitary-indep enden t w ith ev ery sequence y . Note that ev ery sequence x that is the charact eristic sequ en ce of a c.e. set is C- logarithmic. This follo ws from the observ ation that, for ev ery n , the initial segmen t x ↾ n can b e constructed giv en the n umb er of 1’s in x ↾ n (an in formation which can b e written with log n b its) and the finite description of the en umerator of the set represented by x . If a sequence is H -trivial then it is C-logarithmic, but the con ve rse p robably do es n ot hold. In brief, the notions of indep endence and finitary-indep endence are relev ant for strings ha ving complexit y ab o ve that of H -trivial sequences, resp ectiv ely C-logarithmic s equences. The cases of ind ep endent (fin itary-indep endent) pairs ( x, y ), wh ere at least one of x and y is H -trivial (resp ectiv ely , C-logarithmic) will b e r eferred to as trivial indep endenc e . 7 Remark 6 Some desirable pr op erties of the in dep enden ce relat ion are: P1. Symmetry: x is in dep endent w ith y iff y is indep end en t with x . P2. Robustness u nder t yp e of complexit y (plain or prefix-free). P3. If f is a T uring reduction, except for some s p ecial cases, x and f ( x ) are dep enden t (“indep endence cannot b e created”). P4. F or e v ery x , th e set of sequences that are dep enden t with x is small (i.e., it has measure zero). Clearly b oth the indep end ence and the finitary-indep endence relatio n s satisfy P1. Th ey also satisfy P2, as we noted in Remark 3. It is easy to see that the ind ep endence relatio n satisfies P3, wh en ev er we require that the initial segmen ts of x and f ( x ) hav e plain complexit y ω (log n ) (b ecause C x ( f ( x ) ↾ n ) = O (log n ), wh ile C ( f ( x ) ↾ n ) = ω (log n )). W e shall see that the finitary-indep end ence rela- tion satisfies P3 under some stronger assumptions for f and f ( x ) (see Section 4.1 and in particular Th eorem 4.8). W e do not kno w whether the indep end ence relation satisfies P4. Th eorem 3.3 sho ws that the finitary-indep end en ce relat ion satisfies P4. 2.1 Prop erties of independen t and finitary-indep enden t sequences The follo wing s im p le prop erties of finitary-ind ep endent sequences are tec hn ically usefu l in some of the next pro ofs. Prop osition 2.4 (a) Two se quenc es x and y ar e finitary-indep e ndent ⇔ for al l n and m , C ( x ↾ n | y ↾ m ) ≥ C ( x ↾ n ) − O (log n + log m ) . (b) Two se quenc es x and y ar e finita ry-indep endent if and only if for al l n , C ( x ↾ n y ↾ n ) ≥ C ( x ↾ n ) + C ( y ↾ n ) − O (log ( n )) . (c) Two se quenc es x and y ar e finitary-indep endent if and only if f or al l n , C ( x ↾ n | y ↾ n ) ≥ C ( x ↾ n ) − O (log ( n )) . (d) If x and y ar e not finitary-indep endent, then for every c onstant c ther e ar e infinitely many n such that C ( x ↾ n y ↾ n ) < C ( x ↾ n ) + C ( y ↾ n ) − c log n . (e) If x and y ar e not finitary-indep endent, then for every c onstant c ther e ar e infinitely many n such that C ( x ↾ n | y ↾ n ) < C ( x ↾ n ) − c log n . Pr o of . W e use the follo w ing inequalities whic h hold for ev ery strings x and y (they follo w from the Sym metry of Information Equation (3)): C ( xy ) ≥ C ( x ) + C ( y | x ) − O (log | x | + log | y | ) , (4) 8 and C ( xy ) ≤ C ( x ) + C ( y | x ) + O (log | x | + log | y | ) . (5) (a)“ ⇒ ” C ( x ↾ n | y ↾ m ) ≥ C ( x ↾ n y ↾ m ) − C ( y ↾ m ) − O (log n + log m ) (b y (5)) ≥ C ( x ↾ n ) + C ( y ↾ m ) − C ( y ↾ m ) − O (log n + log m ) (b y indep endence) = C ( x ↾ n ) − O (log n + log m ) . “ ⇐ ” C ( x ↾ n y ↾ m ) ≥ C ( y ↾ m ) + C ( x ↾ n | y ↾ m ) − O (log n + log m ) (by (4)) ≥ C ( y ↾ m ) + C ( x ↾ n ) − O (log n + log m ) (b y h yp othesis) . (b) “ ⇒ ” T ak e n = m . “ ⇐ ” Sup p ose n ≥ m (the other case can b e handled similarly). C ( x ↾ n y ↾ m ) ≥ C ( y ↾ m ) + C ( x ↾ n | y ↾ m ) − O (log ( n ) + log ( m )) (b y (4)) ≥ C ( y ↾ m ) + C ( x ↾ n | y ↾ n ) − O (log( n ) + log( m )) ≥ C ( y ↾ m ) + C ( x ↾ n ) − O (log ( n ) + log ( m )) (b y (a)) . (c) This f ollo w s fr om (b) with a similar pro of as for (a). (d) S upp ose that for some constan t c the inequalit y holds only for finitely many n . Then one ca n c ho ose a constant c ′ > c for whic h th e opp osite inequalit y holds for eve ry n , whic h b y (b ) would imply the fi n itary-indep enden ce of x and y . (e) F ollo ws from (c), in a similar wa y as (d) follo ws f rom (b).  Prop osition 2.5 If the se qu enc es x and y ar e indep endent, then they ar e also finitary- indep endent. Pr o of . S upp ose x and y are not finitary-indep en den t. By Prop osition 2.4 (e), for ev ery constan t c th ere are infinitely man y n such that C ( x ↾ n | y ↾ n ) < C ( x ↾ n ) − c · log n . T aking in to account inequalit y (2), we ob tain C y ( x ↾ n ) < C ( x ↾ n ) − ( c − 3) log n , for infi nitely many n , which con tradicts that x and y are indep en den t.  Prop osition 2.6 If dim( x ) = σ and ( x, y ) ar e finitary-indep endent, then dim( x XOR y ) ≥ σ . Pr o of . Note that C ( x ↾ n | y ↾ n ) ≤ C (( x XOR y ) ↾ n ) + O (1), for all n (this h olds for all sequences x and y ). Supp ose th er e exists ǫ > 0 suc h that d im( x XOR y ) ≤ σ − ǫ . It follo ws that, for infi nitely many n , C (( x X OR y ) ↾ n ) ≤ ( σ − ǫ ) n . Th en C ( x ↾ n | y ↾ n ) < C (( x XOR y ) ↾ n ) + O (1) < ( σ − ǫ ) n + O (1) for in finitely man y n. 9 By the finitary-indep end ence of ( x, y ), C ( x ↾ n ) ≤ C ( x ↾ n | y ↾ n ) + O (log n ) ≤ ( σ − ǫ/ 2) n + O (1), i.o. n , whic h con tradicts the fact th at dim( x ) = σ .  Prop osition 2.7 (a) If x i s r andom and ( x, y ) ar e finitary-indep endent, then ( y , x X OR y ) ar e finitary-indep endent. (b) If x is r andom and ( x, y ) ar e indep e ndent, then ( y , x X OR y ) ar e i ndep endent. Pr o of . W e pro ve (a) ( (b) is similar). Su pp ose that y and x XOR y are not finitary- indep end en t. Then fo r ev ery constant c , there are infinitely many n , suc h that C (( x X OR y ) ↾ n | y ↾ n ) < C (( x X OR y ) ↾ n ) − c log n . Note that if a p rogram can pro- duce ( x X OR y ) ↾ n given y ↾ n , then by doing an extra b it wise XOR with y ↾ n it will pro duce x ↾ n . Thus, C ( x ↾ n | y ↾ n ) < C (( x XOR y ) ↾ n | y ↾ n ) + O (1) for all n . Combining with th e first inequalit y , for every constan t c and for infinitely many n w e ha ve : C ( x ↾ n | y ↾ n ) < C (( x XOR y ) ↾ n ) − c log n + O (1) < n − c log n + O (1) < C ( x ↾ n ) + 2 log n − c log n + O (1) = C ( x ↾ n ) − ( c − 2) log n + O (1) . This cont radicts the fact that x and y are fi nitary-indep end ent.  Prop osition 2.8 Ther e ar e se quenc es x, y , and z such that ( x, y ) ar e indep endent, ( x, z ) ar e indep endent, but ( x, y ⊕ z ) ar e not finitary-indep endent. Pr o of . T ak e y and z tw o sequences that are random relativ e to eac h other, and let x = y X OR z . Th en ( x, y ) are ind ep endent, and ( x, z ) are indep enden t, by Prop osition 2.7. On the other hand note that dim( y X O R z ) = 1 (by Prop osition 2.6) and C (( y X OR z ) ↾ n | ( y ⊕ z ) ↾ 2 n ) < O (1). Consequent ly , for ev ery constant c and for almost every n , C (( y XOR z ) ↾ n | ( y ⊕ z ) ↾ 2 n ) < C (( y X O R z ) ↾ n ) − c (log n + log 2 n ), and th us, ( y X OR z , y ⊕ z ) are n ot finitary-indep enden t.  In Remark 5, w e ha ve listed several t yp es of sequences that are indep enden t or fin itary- indep end en t with an y other sequence. The next result go es in the opp osite direction: it exhibits a pair of sequences that can not b e finitary-indep enden t (and th u s not indep en- den t). Prop osition 2.9 [Ste07] If x and y ar e left c.e. se q u enc es, dim( x ) > 0 , and dim( y ) > 0 , then x and y ar e not finitary-indep endent. Pr o of . F or eac h n , let cm x ( n ) = min { s | x s ↾ n = x ↾ n } and cm y ( n ) = min { s | y s ↾ n = y ↾ n } (the con v ergence m o duli of x and, resp ectiv ely , y ). Without loss of generalit y we can assume that cm x ( n ) > cm y ( n ), for infinitely man y n . F or eac h n satisfying the in equalit y , y ↾ n can b e computed from x ↾ n as follo ws. First compute s = cm x ( n ) (which can b e done b ecause 10 x ↾ n is kno wn) and output y s ↾ n . Consequ ently , for infin itely many n , C ( y ↾ n | x ↾ n ) < O (1). On the other hand, since dim( y ) > 0, there exists a constan t c su c h th at C ( y ↾ n ) ≥ c · n , for almost ev ery n . Cons equ en tly , x and y are not finitary-indep end en t.  3 Examples of indep enden t and finitary-indep enden t se- quences W e giv e examples of p airs of s equ ences that are indep end en t or finitary-indep enden t (other than the trivial examples from Remark 5). Theorem 3.1 L et x b e a r andom se quenc e and let y b e a se quenc e that i s r andom r elative to x . Then x and y ar e indep endent. Pr o of . Since y is random relativ e to x , for all n , C x ( y ↾ n ) > n − 2 log n − O (1) ≥ C ( y ↾ n ) − 2 log n − O (1). The v an Lambalg en Theorem [vL90] implies that x is random relativ e to y as w ell. Therefore, in the same w a y , C y ( x ↾ n ) > n − 2 log n − O (1) ≥ C ( x ↾ n ) − O (log n ).  F rom T heorem 3.1 w e can easily deriv e examples of pairs ( x, y ) that are ind ep endent and whic h hav e constru ctiv e Hausdorff dimens ion ǫ , for eve ry rational ǫ > 0. F or exa mple, if we start with x a nd y that are random with resp ect to eac h other and build x ′ = x (1) 0 x (2) 0 . . . (i.e., we insert 0s in the eve n p ositions) and similarly build y ′ from y , then x ′ and y ′ ha ve constructiv e Hausdorff d imension equal to 1 / 2 and are indep end en t (b ecause C x ′ ( y ′ ↾ n ) and C x ( y ↾ ( n/ 2)) are w ithin a constan t of eac h other, as are C ( y ′ ↾ n ) and C ( y ↾ ( n/ 2))). The p airs of sequences from Theorem 3.1 (plus those deriv ed from there as ab o v e) and those from Remark 5 are the only examples of ind ep endent sequences that w e kno w . Th u s, cu rren tly , w e ha ve examples of indep end en t pairs ( x, y ) only for the case when x has m aximal p refix- free complexity (i.e., x is random) or x is obtained via a straigh tforw ard transformation as ab o ve from a random sequ ence, and for the case wh en x h as minimal p refix-free complexit y (i.e., x is H -trivial). W e b eliev e th at for ev ery x , there are sequences y indep end en t with it, and moreo ver we b eliev e that the set of sequences indep endent with x has measur e one. F or finitary-indep en d ence these facts are tr u e. Theorem 3.2 L et x b e an arbitr ary se quenc e and let y b e a se quenc e that is r andom c onditione d by x . Th en x and y ar e finitary-indep endent. Pr o of . Supp ose x and y are not finitary-indep endent. T hen there are infin itely many n with C ( y ↾ n | x ↾ n ) < C ( y ↾ n ) − 5 log n . Cons id er a constan t c 1 satisfying C ( y ↾ n ) < n + c 1 , for all n . W e get (und er our assum p tion) that, for infi nitely man y n . C ( y ↾ n | x ↾ n ) < n − 5 log n + c 1 . Then, by inequalit y 2, for in finitely man y n , C x ↾ n ( y ↾ n ) < n − 3 log n + c + c 1 . Note that that f or eve r y n and eve ry m ≥ n , C x ↾ m ( y ↾ n ) < C x ↾ n ( y ↾ n ). Th us, f or infinitely man y n and for all m > n , C x ↾ m ( y ↾ n ) < n − 3 log n + ( c + c 1 ) . (6) 11 On the other hand , y is random conditioned b y x . Therefore, for all n , H x ( y ↾ n ) > n − O (1). Let U ′ b e the un iv ersal mac hine underlying the complexit y H ( · ) and let p ∗ b e the shortest program suc h that U ′ x ( p ∗ ) = y ↾ n (if ther e are ties, tak e p ∗ to b e the lexico graphically smallest among the t ying pr ograms). Let m ( n ) = min ( n, u s e( U ′ x ( p ∗ ))). Note th at, for all n , H x ( y ↾ n ) = H x ↾ m ( n ) ( y ↾ n ). It f ollo ws that, for every n , H x ↾ m ( n ) ( y ↾ n ) = H x ( y ↾ n ) > n − O (1). Recall that for ev ery strings u and v , C v ( u ) > H v ( u ) − 2 log | u | − O (1). Thus, for ev ery n , C x ↾ m ( n ) ( y ↾ n ) > n − 2 log n − O (1) . (7) Inequalities (6) and (7) are contradicto r y .  Theorem 3.3 F or every x , the set { y | y finitary-indep endent with x } has me asur e one. Pr o of . By the pr evious result, the set in the statemen t of the theorem con tains the set { y | y rand om conditioned by x } which has measure one.  Th us there are m any (in the measur e-theoretical sense) pairs of sequ ences that are finitary-indep enden t. But is it p ossible to hav e such pairs satisfying a given constraint ? W e consider one instance of this general issue. Prop osition 3.4 If x is a r andom se quenc e then ther e ar e y and z such that ( y , z ) ar e finitary-indep endent and x = y X OR z . Pr o of . T ake a sequence y finitary-indep end en t with x . Th en, by Prop osition 2.7, y and ( x XOR y ) are finitary-ind ep endent. By taking z = x XOR y , it follo w s that x = y X O R z , with y and z fi nitary-indep end en t.  4 Effectiv e constructions of finitary-indep endent sequences The examples of (finitary-) ind ep endent sequences that w e ha ve pro vided so far are exis- ten tial (i.e. , non-constructiv e). In this section we in vestig ate to w hat extent it is p ossible to effectiv ely c onstruct suc h se quences. W e sh o w some imp ossibility r esults and there- fore we fo cu s on the weak er type of indep endence, finitary-indep end ence (clea rly , if it is not p ossible to pro d uce a p air of sequences that are fin itary-indep endent, th en it is also not p ossible to pr o duce a pair of sequences that are ind ep enden t). Since a C-logarithmic sequence is fin itary-indep endent with an y other sequence, the issue of constructibilit y is in - teresting if we also r equire that the sequ ences h av e complexit y abov e that of C-logarithmic sequences (see Remark 5). Su c h sequences are of course non-computable, an d therefore the w hole issue of constructibilit y app ears to b e a mo ot p oin t. How ever this is not s o if we assume that w e already ha ve in h an d one (or sev eral) non-computable sequence(s), and we w ant to build additional sequences that are finitary-indep enden t. Informally sp eaking, we in vestig ate the follo wing questions: 12 Question (a) Is it p ossible to effectiv ely co nstruct f rom a s equ ence x another sequ ence y (finitary-) in dep endent with x , where the indep endence is not trivial (recall Remark 5)? This question has t wo v arian ts dep ending on whether we seek a unif orm pro cedur e (i.e., one pr o cedure that w orks for all x ), or whether we allo w the pro cedure to dep end on x . Question (b) Is it p ossible to effectiv ely construct from a sequence x t w o sequences y and z that are (finitary-) indep end en t, where the indep end ence is not trivial? Again, there are un iform and non-un iform v ariants of this qu estion. W e analyze these questions in Section 4.1. S imilar questions for the case when the input consists of t wo sequences x 1 and x 2 are tac kled in S ection 4.2. 4.1 If w e ha v e one source W e fir st consider th e u niform v ariant of Qu estion (a): Is there a T ur ing reduction f suc h that f or all x ∈ { 0 , 1 } ∗ , ( x, f ( x )) are (fin itary-) indep enden t? W e ev en relax the require- men t and demand that f sh ould achiev e this ob jectiv e only if x has p ositive constructive Hausdorff dimension (this only make s the follo wing imp ossibilit y results stronger). As discussed ab o v e, we first eliminate some trivial instances of th is question. Without an y requirement on the algorithmic complexit y of the desired f ( x ), the answ er is trivially YES b ecause we can tak e f ( x ) = 0 ω (or an y other computable sequence). Ev en if w e only require that f ( x ) is not computable, then the an s w er is still trivially YES b ecause w e can mak e f ( x ) to b e C-log arithmic. F or example, consid er f ( x ) = x (1) x (2)0 x (3)000 . . . x ( k ) 0 . . . 0 | {z } 2 k − 1 − 1 . . . . Then f ( x ) is C-log arithmic, b ut not computable pro vid ed x is not compu table, and ( x, f ( x )) are fin itary-ind ep endent simp ly b ecause f ( x ) is C-logarithmic. As noted ab ov e, the question is interesting if we require f ( x ) to hav e some “significant ” amoun t of randomness when ever x has some “significant” amount of randomness. W e exp ect th at in this case the answer should b e negativ e, b ecause, in tuitiv ely , one sh ould not b e able to pro duce indep endence (this is prop ert y P3 in Remark 6). W e consider tw o situations dep ending on t w o d ifferen t meanings of the conce pt of “significan t” amoun t of randomn ess. Case 1: W e require that f ( x ) is n ot C-logarithmic. W e do not solv e the question, but w e sh o w that ev ery r ed uction f that p oten tially d o es the job must hav e non-p olynomial use. Prop osition 4.1 L et f b e a T uring r e duction. F or every se q uenc e x , if the function use x f ( n ) is p olynomial ly b ounde d, then x and f ( x ) ar e not finitary-indep endent, unless one of them is C-lo garithmic. 13 Pr o of . Let y b e f ( x ). Then for ev ery n , let m ( n ) = max k ≤ n use x f (1 n )). Then y ↾ n dep ends only on x ↾ m ( n ) and m ( n ) is p olynomial in n . Then C ( y ↾ n | x ↾ m ( n )) ≤ O (log n ). If x and y were finitary-indep endent, then C ( y ↾ n ) ≤ C ( y ↾ ( n ) | x ↾ m ( n )) + O (log n + log m ( n )) ≤ O (log ( n )) + log( m ( n )) ≤ O (log n ), for al l n , i.e., y would b e C-loga r ithmic .  Case 2: W e require that f ( x ) has complexit y ju st ab o v e t hat of C-loga rithmic se- quences (in the s en se b elo w). W e sho w th at in this case, the answ er to the uniform v ariant of Qu estion (a) is negativ e: t here is no such f . The follo w ing definition in tr o duces a class of sequences h a ving complexit y ju s t ab o ve that of C -logarithmic sequences. Definition 4.2 A se quenc e x is C-sup erlo garithmic if for every c onstant c > 0 , C ( x ↾ n ) > c log n , for almost every n . The next pr o ofs use the follo wing facts. F act 4.3 (V ariant of Theorem 3.1 in [NR06]) F or al l r ationals 0 ≤ α < β < 1 , and for every set S that is infinite and c omputable, ther e exists a se quenc e x such that d im( x ) = α and for al l wtt-r e ductions f , ei ther f ( x ) do es not exist or C ( f ( x ) ↾ n ) ≤ β n , for i nfinitely many n in S . F act 4.4 (V ariant of Theorem 3.1 in [BDS07]) F or every T uring r e duction h , for al l r atio- nals 0 < α < β < 1 , and for every set S that is infinite and c omputable, ther e is a se quenc e x with dim( x ) ≥ α such that either h ( x ) do es not exist or C ( h ( x ) ↾ n ) < β n , f or infinitely many n in S . F act 4.5 (Theorem 4.15 in ([Zim07]) F or any δ > 0 , ther e exist a c onstant c , a set S that is infinite and c omputable, and a truth-table r e duction g : { 0 , 1 } ∞ × { 0 , 1 } ∞ → { 0 , 1 } ∞ (i.e., g is a T uring machine with two or acles) with the fol lowing pr op erty: If the input se quenc es x and y ar e finitary-indep endent and satisfy C ( x ↾ n ) > c · log n and C ( y ↾ n ) > c · log n , for almost every n , then the output z = f ( x, y ) satisfies C ( f ( x, y ) ↾ n ) > (1 − δ ) · n , for almost every n in S . Theorem 3.1 in [NR06] is for S = N (and is stronger in that α = β ) but its pro of can b e mo dified in a straigh tforw ard manner to yield F act 4.3. Theorem 3. 1 in [BDS07] is also for S = N and can also b e mo dified in a simp le manner – usin g F act 4.3 – to yield F act 4.4. W e can n o w state the imp ossibilit y results related to Case 2 . T o simp lify the stru ctur e of quantifiers in the statement of the f ollo w ing result, we p osit here th e follo win g task for a fun ction f mappin g sequences to s equ ences: T AS K A: for ev ery x ∈ { 0 , 1 } ∞ with dim( x ) > 0, the follo w ing should hold: (a) f ( x ) exists. (b) f ( x ) is C-su p erlogarithmic. (c) x and f ( x ) are finitary-indep enden t. 14 Theorem 4.6 Ther e is no T uring r e duction f that satisfies T A SK A. Pr o of . S upp ose there exists f satisfying (a), (b) and (c) in T ASK A. Let S b e the infinite, computable set and let g b e the tru th-table red uction promised b y F act 4.5 for δ = 0 . 3. Let h b e the T uring reduction h ( x ) = g ( x, f ( x )). L et x ∗ b e th e sequen ce promised b y F act 4.4 for α = 0 . 5, β = 0 . 6, and the ab o v e set S and T uring reduction h . On one hand, by F act 4.4, C ( h ( x ∗ ) ↾ n ) < 0 . 6 n , for infinitely many n ∈ S . O n the other han d , by F act 4.5, C ( h ( x ∗ ) ↾ n ) > 0 . 7 n , for almost ev ery n ∈ S . W e ha ve reac hed a cont radiction.  W e next consider the uniform v arian t of Question (b). First w e remark, that b y v an Lam b algen Theorem [vL90], if the sequence x is random, then x eve n and x odd are rand om relativ e to eac h other (where x odd is x (1) x (3 ) x (5) . . . and x eve n is x (2) x (4) x (6) . . . ). Thus, x eve n and x odd are certainly indep end ent. Kautz [Kau03] has sho wn a muc h more general result b y examining the sp littings of sequences obtained using b ounded Kolmog oro v-Lo vela nd selecti on ru les. 1 He sho wed that if x is a random sequence, x 0 is the s ubsequence of x obtained by concatenating the bits of x chosen b y an arbitrary b ound ed Kolmog orov-Lo vela n d selection rule, and x 1 consists of the bits of x that w ere not selected b y the selection rule, then x 0 and x 1 are random with resp ect to eac h other (and thus indep endent). W e sho w that the similar result for sequences with constructive Hausdorff d imension σ ∈ (0 , 1) is not v alid. In fact, our next r esu lt is stronger, and essent ially give s a negativ e answ er to the un iform v ariant of Question (b). W e p osit the follo wing task for tw o functions f 1 and f 2 mapping sequences to sequences: T AS K B: for ev ery x ∈ { 0 , 1 } ∞ with d im( x ) > 0, the follo w ing should hold: (a) f 1 ( x ) and f 2 ( x ) exist, (b) f 1 ( x ) and f 2 ( x ) are C -sup erlogarithmic, (c) f 1 ( x ) and f 2 ( x ) are fi nitary-indep end en t. Theorem 4.7 Ther e ar e no T uring r e ductions f 1 and f 2 satisfying T ASK B. Pr o of . Similar to the pro of of Theorem 4.6.  The non-u niform v arian ts of Questions (a) and (b) remain op en. In the particular case when f is a wtt-reduction, we pr esen t imp ossibilit y results analo gous to those in Theorem 4.6 and Th eorem 4.7. The pro ofs are similar, with th e difference that w e use F act 4.3 instead of F act 4.4. 1 A Kolmogoro v- Lo veland selec tion ru le is an effective pro cess for selecting bits from a sequen ce. Infor- mally , it is an iterative pro cess and at each step, based on the bits that hav e b een already read, a new bit from the sequence is chosen to b e read and (b efore t h at b it is actually read) the decision on whether th at bit is sel ected or not is t aken. A b ounde d Kolmogoro v- Lo veland selection rule satisfies a certain req uirement of monotono city for deciding th e selected bits, see [Kau03]. 15 Theorem 4.8 F or al l r ational σ ∈ (0 , 1) , ther e exists dim( x ) = σ such that for every wtt-r e duction f , at le ast one of the fol lowing holds true: (a) f ( x ) do es not exist, (b) f ( x ) is not finitary-indep endent with x , (c) f ( x ) is not C-sup erlo garithmic. Theorem 4.9 F or al l r ational σ ∈ (0 , 1) , ther e exists x with d im( x ) = σ such that for every wtt-r e ductions f 1 and f 2 , at le ast one of the fol lowing holds true: (a) f 1 ( x ) do es not exist or f 2 ( x ) do es not exist, (b) f 1 ( x ) and f 2 ( x ) ar e not finitary-indep endent, (c) f 1 ( x ) is not C-sup e rlo garithmic or f 2 ( x ) is not C-sup e rlo garithmic. Theorem 4.9 has an interesting implication regarding sequences with constructiv e Hausd orff dimension in the interv al (0 , 1). Sup p ose, for example that we wan t to construct a sequence with constructiv e Hausdorff dimension 1/2. The first idea that comes to mind is to tak e a random sequence x = x (1) x (2) . . . and either consider the sequence y = x (1)0 x (2)0 . . . (we insert 0s in all ev en p ositions) or the sequence z = x (1) x (1) x (2) x (2) . . . (w e double ev ery bit). The sequ ences y and z hav e constructive Hausd orff d imension 1/2. Th eorem 4.9 sho ws , roughly sp eaking, that there are sequences with dimension strictly b et w een 0 and 1, wh ere th e p artial randomn ess is du e n ecessarily to one of the tw o metho ds stat ed ab o ve . F ormally , for ev ery rational σ ∈ (0 , 1), there is a sequence x with d im( x ) = σ so th at no matter what wtt metho d we use for selecting from x t wo subsequences, either one of the resulting su bsequences has low complexit y or the tw o resulting subsequences are not indep end en t. 4.2 If w e ha v e tw o sources W e ha ve seen some limits on the p ossibilit y of constructing a finitary-indep enden t sequences starting fr om one sequence. What if we are giv en tw o finitary-ind ep endent sequences: is it p ossible to constru ct from th em more fin itary-indep endent sequ ences? First we observ e that if x and y are t wo indep en d en t sequ ences and g is an arbitrary T u ring reduction, then it do es not necessarily follo w that x an d g ( y ) are indep en d en t (as one ma y exp ect). On the other hand it d o es follo w that x and g ( y ) are fi nitary-indep end en t. Prop osition 4.10 [Ste07] Ther e ar e two indep endent se quenc e s x and y and a T uring r e duction g such that x and g ( y ) ar e not i ndep endent. Pr o of . Let z b e a ran d om sequence and let u, v , and w b e sequences such that z = u ⊕ v ⊕ w . By v an Lam b algen Theorem [vL90], eac h of the sequences u, v , an d w are random relativ e 16 to the j oin of the other t wo. W e defin e the sequences x and y as follo w s: x (2 n ) = u ( n ) , for all n ∈ N x ( m ) = v ( m ) , for ev ery m that is n ot a p o we r of 2 y (2 n ) = u ( n ) , for all n ∈ N y ( m ) = w ( m ) , for ev ery m that is not a p o w er of 2 Claim 4.11 The se quenc es x and y ar e indep endent. Pr o of . Supp ose x and y are not indep end ent. Then, similarly to Prop osition 2.4 (e), for infinitely many n , C x ( y ↾ n ) < C ( y ↾ n ) − 7 log n . Then C u ⊕ v ( w ↾ n ) ≤ C u ⊕ v ( y ↾ n ) + 2 log n + O (1) (b ecause w ↾ n and y ↾ n differ in only log n bits) ≤ C x ( y ↾ n ) + 2 log n + O (1) (b ecause queries to x can b e replaced b y queries to u and v ) ≤ C ( y ↾ n ) − 7 log n + 2 log n + O (1) , for infin itely many n ≤ C ( w ↾ n ) + 2 log n − 7 log n + 2 log n + O (1) = C ( w ↾ n ) − 2 log n + O (1) ≤ n − 3 log n + O (1) . This cont radicts that w is rand om with resp ect to u ⊕ v .  It is easy to define a T urin g r eduction g such that g ( y ) = u . Notice that C x ( u ↾ n ) = O (log n ), b ecause u is man y-one reducible to x . O n the other hand C ( u ↾ n ) ≥ n − 2 log n + O (1), for ev ery n , b ecause u is random. Therefore x and g ( y ) are n ot indep enden t.  W e do not know if the facts that x and y are fi nitary-indep end en t and g is a T urin g reduction, imp ly that x and g ( y ) are finitary-indep en den t This w ou ld sho w th at finitary- dep enden cy cannot b e created. The follo wing wea ker resu lt holds. Prop osition 4.12 If x and y ar e indep endent, and g is a T uring r e duction, then x and g ( y ) ar e finitary-indep endent (pr ovide d g ( y ) exists). Pr o of . Since x and y are in dep endent, there exists a constant c su c h that for all n, C y ( x ↾ n ) ≥ C ( x ↾ n ) − c log n . Supp ose that x and g ( y ) are not fi nitary-indep end en t. Then there are infi nitely man y n suc h that C ( x ↾ n | g ( y ) ↾ n ) < C ( x ↾ n ) − ( c + 4) log n . S ince C y ( x ↾ n ) ≤ C ( x ↾ n | g ( y ) ↾ n ) + 2 log n + O (1), it w ould follo w that, for infinitely man y n , C y ( x ↾ n ) ≤ C ( x ↾ n ) − ( c + 1) log n, whic h con tradicts the fir s t inequalit y .  17 Corollary 4.13 Ther e ar e se quenc es that ar e finitary-indep endent b u t not indep endent. Pr o of . The sequ ences x and g ( y ) from Prop osition 4.10 are not indep enden t, but they are finitary-indep enden t by Prop osition 4.12.  As w e men tioned, w e d o not kno w if Pr op osition 4.12 can b e strengthened to hold if x and y are finitary-indep enden t. How ev er, for suc h x an d y , th ere exists a simple p ro cedure that starting w ith the p air ( x, y ), pro duces a new pair of finitary-indep enden t sequences. Namely , w e tak e the pair ( x, y odd ). Prop osition 4.14 If x and y ar e finitary-indep endent, then x and y odd ar e finitary- indep endent. Pr o of . Sup p ose that for ev ery constant c there are infin itely man y n suc h that C ( x ↾ n | y odd ↾ n ) < C ( x ↾ n ) − c · log n . Note that, for all n , C ( x ↾ n | y ↾ 2 n ) ≤ C ( x ↾ n | y odd ↾ n ) + O (1). Our assumption implies that for ev ery constan t c th ere are infinitely m an y n such that C ( x ↾ n | y ↾ 2 n ) < C ( x ↾ n ) − c log n + O (1). By Prop osition 2.4, (a), this cont r adicts the fact that x and y are fin itary-indep endent.  The next issue that w e study is whether giv en a pair of (finitary-)indep endent strings ( x, y ), it is p ossible to effectiv ely pro duce an other string that is (finitary-)indep end en t with b oth x and y . W e giv e a p ositiv e answer for the case when x and y are b oth r andom. The similar question f or non-random x and y remains op en (but see Section 4.3). Theorem 4.15 Ther e exists an effe ctive tr ansformation f w i th p olynomial ly-b ounde d use such that if x and y ar e r andom and indep endent (r esp e ctive ly finitary-indep endent), then f ( x, y ) is indep endent (r esp e ctively, finitary-indep endent) with b oth x and y , and the inde- p endenc e i s not trivial (r e c al l R emark 5). Remark: C on trast with Pr op osition 4.1, w h ere w e ha ve shown that for ev ery x , for ev- ery effectiv e transformation f with p olynomially-b ound ed use, x and f ( x ) are not finitary- indep end en t. Pr o of . W e tak e f ( x, y ) = x X OR y and tak e into accoun t Prop osition 2.7.  4.3 Pro ducing indep endence: the finite case An int eresting issue is w hether giv en as input sev er al sequences that are (fi nitary-) inde- p endent, there is an effectiv e wa y to construct a sequence that is (fi n itary-) in d ep endent with eac h sequence in th e inp ut (and the in d ep endence is n ot trivial). A result of this t yp e is obtained for the case when th e inpu t consists of t wo random sequences x and y in Theorem 4.15. W e do n ot kno w if in Theorem 4.15 we can remo v e the assumption that x and y are random. 18 In what follo ws w e w ill consider the simpler case of strings. In this setting w e are able to giv e a p ositiv e answ er for the situation when we start with three 2 input strings that are indep end en t (and not n ecessarily random). First we d efine the analogue of ind ep endence for strings. Definition 4.16 L et c ∈ R + and k ∈ N . We say that strings x 1 , x 2 , . . . , x k in { 0 , 1 } ∗ ar e c -indep endent if C ( x 1 x 2 . . . x k ) ≥ C ( x 1 ) + C ( x 2 ) + . . . + C ( x k ) − c (log | x 1 | + log | x 2 | + . . . + log | x k | ) . The m ain result of th is sectio n is the follo wing theorem, whose pro of d ra ws from the tec hniques of [Zim07]. Theorem 4.17 F or al l c onstants σ > 0 and σ 1 ∈ (0 , σ ) , ther e exists a c omputable function f : { 0 , 1 } ∗ × { 0 , 1 } ∗ × { 0 , 1 } ∗ → { 0 , 1 } ∗ with the fol lowing pr op erty: F or every c ∈ R + ther e exists c ∈ R + such that if the input c onsists of a triplet of c -indep endent strings having sufficiently lar g e length n and plain c omplexity at le ast σ · n , then the output i s c -indep e ndent with e ach e lement in the input triplet and has length ⌊ σ 1 n ⌋ . Mor e pr e cisely, i f (i) ( x, y , z ) ar e c -indep endent, (ii) | x | = | y | = | z | = n , and (iii) C ( x ) ≥ σ · n , C ( y ) ≥ σ · n , C ( z ) ≥ σ · n , then, pr ovide d n is lar ge enough, the fol lowing p airs of strings ( f ( x, y , z ) , x ) , ( f ( x, y , z ) , y ) , ( f ( x, y , z ) , z ) ar e c -indep endent, | f ( x, y , z ) | = ⌊ σ 1 n ⌋ , and C ( f ( x, y , z )) ≥ ⌊ σ 1 n ⌋ − O (log n ) . Before we delv e in to th e pro of, w e establish several preliminary facts. Lemma 4.18 If x 1 , x 2 , x 3 ar e thr e e strings that ar e c -indep endent, then C ( x 1 | x 2 x 3 ) ≥ C ( x 1 ) − ( c + 2)(log | x 1 | + log | x 2 | + log | x 3 | ) − O (1) . Pr o of . The follo w ing inequalities h old for ev er y three strings and in particular f or the strings x 1 , x 2 , and x 3 : C ( x 1 x 2 x 3 ) ≤ C ( x 2 x 3 ) + C ( x 1 | x 2 x 3 ) + 2 log | x 1 | + O (1) , and C ( x 2 x 3 ) ≤ C ( x 2 ) + C ( x 3 ) + 2 log | x 2 | + O (1) . 2 The case when the input consists of t wo indep endent strings remains open. 19 Then C ( x 1 | x 2 x 3 ) ≥ C ( x 1 x 2 x 3 ) − C ( x 2 x 3 ) − 2 log | x 1 | − O (1) ≥ C ( x 1 ) + C ( x 2 ) + C ( x 3 ) − c (log | x 1 | + log | x 2 | + log | x 3 | ) − ( C ( x 2 ) + C ( x 3 ) + 2 log | x 2 | + O (1)) − 2 log | x 1 | − O (1) ≥ C ( x 1 ) − ( c + 2)(lo g | x 1 | + log | x 2 | + log | x 3 | ) − O (1) .  The n ext lemma establishes a combinatorial fact ab out the p ossibilit y of coloring the cub e [ N ] × [ N ] × [ N ] with M colors suc h that ev ery planar r ectangle conta ins all the colors in ab out the same prop ortion. Here N and M are natural num b ers , [ N ] den otes the set { 1 , 2 , . . . , N } , [ M ] denotes the set { 1 , 2 , . . . , M } and a p lanar rectangle is a subset of [ N ] × [ N ] × [ N ] h a ving one of the follo w ing three forms: B 1 × B 2 × { k } , B 1 × { k } × B 2 , or { k } × B 1 × B 2 , wh er e k ∈ [ N ], B 1 ⊆ [ N ] and B 2 ⊆ [ N ]. Lemma 4.19 L et 0 < σ 1 < σ 2 < 1 . F or every n suffici e ntly lar ge, it is p ossible to c olor the cub e [2 n ] × [2 n ] × [2 n ] with M = 2 ⌊ σ 1 n ⌋ c olors in such a way that every planar r e ctangle satisfying k B 1 k = a 2 ⌈ σ 2 n ⌉ and k B 2 k = b 2 ⌈ σ 2 n ⌉ for some natur al numb ers a and b c ontains at most (2 / M ) k B 1 kk B 2 k o c curr enc es of c olor c , for e very c olor c ∈ [ M ] . Pr o of . W e use the pr obabilistic metho d. Let N = 2 n . W e color eac h cell of t he [ N ] × [ N ] × [ N ] cub e with one color c hosen indep enden tly and uniformly at random from [ M ]. F or i, j, k ∈ [ N ], let T ( i, j, k ) b e the random v ariable that d esignates the color of the cell ( i, j, k ) in the cub e. F or ev ery fixed cell ( i, j, k ) and for eve r y fixed color c ∈ [ M ], Prob( T ( i, j, k ) = c ) = 1 / M , b ecause the colors are assigned ind ep endently and unif orm ly at random. Let us first consider some fixed subs ets B 1 and B 2 of [ N ] ha vin g size 2 ⌈ σ 2 n ⌉ , a fixed k ∈ [ N ], and a fixed color c ∈ [ M ]. Let A b e the ev ent “the fraction of occur en ces of c in the planar rectangle B 1 × B 2 × { k } is greater than 2 / M .” Using the Chernoff b ounds, it follo ws that Prob( A ) < e − (1 / 3)(1 / M ) N 2 σ 2 . The same upp er b ound s hold for the p robabilities of the similar ev ents regarding the planar rectangles B 1 × { k } × B 2 and { k } × B 1 × B 2 . Thus, if w e consider the ev en t B “there is some color with a fraction of app earances in one of the three planar rectangles men tioned ab o ve greater than (2 / M )”, then, b y the u nion b oun d, Prob( B ) < 3 M e − (1 / 3)(1 / M ) N 2 σ 2 . (8) The num b er of wa ys to c ho ose B 1 ⊆ [ N ] with k B 1 k = 2 ⌈ σ 2 n ⌉ , B 2 ⊆ [ N ] with k B 2 k = 2 ⌈ σ 2 n ⌉ and k ∈ [ N ] is app ro ximately (ig noring the trun cation)  N N σ 2  ·  N N σ 2  · N , whic h is b oun ded b y e 2 N σ 2 · e 2 N σ 2 (1 − σ 2 ) l n( N ) · e ln N , (9) 20 (w e ha ve used the in equ alit y  n k  < ( en/k ) k ). Clea r ly , for our choic e of M , the right hand side in (9) times the right hand side in (8 ) is less than 1. It means that th ere exists a coloring wh ere no color app ears a fr action larger than (2 / M ) in every planar rectangle with B 1 and B 2 ha ving size exactly 2 ⌈ σ 2 n ⌉ . F or planar rectangles ha ving the sizes of B 1 and B 2 an in teger m ultiple of 2 ⌈ σ 2 n ⌉ , the assertion holds as well b ecause suc h rectangles can b e p artitioned into subrectangles having the size exactly 2 ⌈ σ 2 n ⌉ .  Pr o of of Theorem 4.17. W e tak e n sufficient ly large so that all the follo wing in equ al- ities hold. Let x ∗ , y ∗ and z ∗ b e a triplet of strings of length n satisfying the assump tions in the statemen t. Let N = 2 n and let us consider a constan t σ 2 ∈ ( σ 1 , σ ). By exhaustive searc h we find a coloring T : [ N ] × [ N ] × [ N ] → [ M ] s atisfying the p rop erties in Lemma 4.19. Iden tifying the strings x ∗ , y ∗ and z ∗ with th eir indeces in the lexicographical ordering of Σ n , w e d efine w ∗ = T ( x ∗ , y ∗ , z ∗ ). Note that the length of w ∗ is log M = ⌊ σ 1 n ⌋ , whic h w e denote m . W e w ill sho w that C ( w ∗ | z ∗ ) ≥ m − c ′ log m , for c ′ = 3 c + d + 13, for a constan t d that will b e sp ecified later. Since C ( w ∗ ) ≤ m + O (1), it follo ws that w ∗ and z ∗ are indep end en t. In a similar w ay , it can b e shown that w ∗ and x ∗ are indep enden t, and w ∗ and y ∗ are indep end en t. F or the sak e of obtaining a co ntradiction, supp ose that C ( w ∗ | z ∗ ) < m − c ′ log m . The set A = { w | C ( w | z ∗ ) < m − c ′ log m } has size < 2 m − c ′ log m and, by our assu mption, con tains w ∗ . Let t 1 b e such that C ( x ∗ ) = t 1 and t 2 b e such that C ( y ∗ | z ∗ ) = t 2 . Note that t 1 > σ 2 n . The integ er t 2 is also larger than σ 2 n , b ecause C ( y ∗ | z ∗ ) ≥ C ( y ∗ | z ∗ x ∗ ) − 2 log n − O (1) ≥ C ( y ∗ ) − ( c + 4)(3 log n ) − O (1) ≥ σ n − (3 c + 12) log n − O (1) > σ 2 n . F or the second inequalit y w e ha ve used Lemma 4.18. Let B 1 = { x ∈ Σ n | C ( x ) ≤ t 1 } . Note that B 1 has size b ounded b y 2 t 1 +1 . W e tak e a set B ′ 1 including B 1 ha ving size exactly 2 t 1 +1 . S im ilarly , let B 2 = { y ∈ Σ n | C ( y | z ∗ ) ≤ t 2 } and let B ′ 2 b e a set that includes B 2 and has size exactly 2 t 2 +1 . Let k b e the index of z ∗ in the lexico grap h ical ordering of Σ n . By Lemma 4.19, it follo ws th at for eve r y a ∈ [ M ], k T − 1 ( a ) ∩ ( B ′ 1 × B ′ 2 × { k } ) k ≤ (2 / M ) k B ′ 1 kk B ′ 2 k . Consequent ly , k T − 1 ( A ) ∩ ( B 1 × B 2 × { k } ) k ≤ k T − 1 ( A ) ∩ ( B ′ 1 × B ′ 2 × { k } ) k = P a ∈ A k T − 1 ( a ) ∩ ( B ′ 1 × B ′ 2 × { k } ) k < 2 m − c ′ log m · (2 / 2 m ) k B ′ 1 kk B ′ 2 k = 2 t 1 + t 2 +3 − c ′ log m . Note that giv en z ∗ , m − c ′ log m , t 1 and t 2 , we can enumerate T − 1 ( A ) ∩ ( B 1 × B 2 × { k } ). Since ( x ∗ , y ∗ , z ∗ ) is in this s et, it follo w s that the complexit y of x ∗ y ∗ giv en z ∗ is b oun ded b y the rank of the triplet ( x ∗ , y ∗ , z ∗ ) in a fixed en um er ation of the set and the information needed to p erform the enumeratio n. Thus, C ( x ∗ y ∗ | z ∗ ) ≤ t 1 + t 2 + 3 − c ′ log m + 2 log( m − c ′ log m ) + 2 log t 1 + 2 log t 2 + O (1) ≤ t 1 + t 2 − ( c ′ − 2) log m + 2 log t 1 + 2 log t 2 + O (1) . 21 On the other hand , by th e conditional v ers ion of the Symmetry of Information Equ a- tion (3), there exists a constant d such th at for all str ings u, v , w , C ( uv | w ) ≥ C ( v | w ) + C ( u | uw ) − d (log | uv | ). It follo ws that C ( x ∗ y ∗ | z ∗ ) ≥ C ( y ∗ | z ∗ ) + C ( x ∗ | y ∗ z ∗ ) − d log n − O (1) ≥ t 2 + t 1 − ( c + 2)(3 log n ) − d log n − O (1) = t 1 + t 2 − (3 c + d + 6) log n − O (1) . F or the second inequalit y w e h a ve used Lemma 4.18. Note that t 1 < n + O (1) and t 2 < n + O (1) and m = σ 1 n . Co m b ining the ab o ve inequalities, we obtain ( c ′ − 2) log σ 1 n ≤ (3 c + d + 10 ) log n + O (1). S ince c ′ = 3 c + d + 13, we ha v e ob tained a con tradiction.  5 Ac kno wledgmen ts W e are grateful to An dr´ e Nie s and F rank Steph an for their insigh tfu l commen ts. In partic- ular, Definition 2.1 has emerged after seve r al discussions with Andr´ e, and Prop ositio n 2.9 and Prop osition 4.10 are due to F rank [Ste07]. W e also thank Jan Reimann for his assis- tance with establishing F act 4.3. References [BDS07] L. Bien ve n u , D. Dot y , and F. Stephan. Constructiv e dimension and we ak tru th- table degrees. In Computation and L o gic in the R e al World - Thir d Confer enc e of Computability in E u r op e . Sp r inger-V erlag L e ctur e Notes in Computer Scienc e #4497 , 2007 . T o App ear. Av ailable as T ec hn ical R ep ort arXiv:cs/07010 89 ar [C02] C. S. Calude. Information and R andomness: An Algo rithmic Pe rsp e ctive , 2nd Edition, Revised and Extended, S pringer-V erlag, Berlin, 2002. [CST06] C. Calude, L. Staiger, and S. T erwijn, On p artial r andomness. Annals of Pu r e and Applie d L o gic , 138:20–30, 200 6. [Cha75] G. Chaitin. A theory of program size f ormally iden tical to information theory , Journal of the ACM , 22: 329–3 40, 1975. [Cha82] G. Chaitin. G¨ odel’s Th eorem and Inform ation, International Journal of The o- r etic al P hysics 21: 941–954, 1982. [DH] R. Downey and D. Hirsc hfeldt. Algorithmic r andomness and c omplexity . T o b e published by Springer Verlag. [DHNT06] R. Do w ney , D. Hirsc hfeldt, A. Nies, and S. T erwijn. Calibrating rand omness, The Bul letin of Symb olic L o gic , 12(3):411– 492, 2006. 22 [GV04] P . Gr ¨ un w ald and P . Vitan yi. Shannon information and Kolmogoro v complexit y , 2004. CORR T ec h nical r ep ort arxiv:cs.IT/041000 2 , revised May 2006. [Kau03] S.M. Kautz. Indep endence prop erties of algorithmicall y random sequences, 2003. CORR T ec h nical Rep ort arXiv:cs/03 01013 . [Lev84] L. Levin. Randomness conserv ation inequalities: information and indep en d ence in mathematical theories. Informat ion and Contr ol , 61(1 ), 1984. [Lut03] J. Lu tz. The d imensions of individual strings and sequences, Info rmation and Contr ol , 187:49 –79, 2003. [Ma y02] E. Ma yordomo. A Kolmogoro v complexit y c h aracteriza tion of constru ctive Hausdorff dimension, Information Pr o c essing L etters , 84:1 –3, 200 2. [ML66] P . Martin-L¨ of. The d efinition of random sequences, Information and Contr ol , 9:602– 619, 1966 . [Nie05] A. Nies. Lo wness prop erties and r andomness, A dvanc es in Mathematics , 197:27 4–305, 2005 . [NR06] A. Nies and J . Reimann. A lo wer cone in the w tt degrees of non-in tegral effectiv e dimension, In Pr o c e e dings of IMS workshop on Computational Pr osp e cts of Infinity , S ingap ore, 2006. T o app ear. [Rei04] J . Reimann. Comp utabilit y and fractal dimension, T ec hnical r ep ort, Univ er- sit¨ at Heidelb erg, 2004. Ph.D. thesis. [Ry a84] B. Ryabk o. Co ding of com binatorial sources and Hausdorff dimens ion, Doklady Akad e mii Nauk SSR , 277:1066 –1070, 1984. [Sol75] R. Solo v ay . Draft of a pap er (or series of pap ers) on Ch aitin’s work, 1 975. unpu blished man uscrip t, I BM Thomas J. Watson Reserac h Cen ter, 215 pp . [Sta93] L. Staiger. Kolmogo r o v complexit y and Hausdorff dimension, Inform. and Com- put. 103 (199 3) 159-194 . [Sta05] L. Staiger. Constructiv e d imension equals K olmogoro v complexit y , Informa tion Pr o c essing L etters , 93:149–1 53, 2005. Preliminary v ersion: Researc h Rep ort CDMTCS-210, Univ. of Auc kland , January 2003 . [Ste07] F. Stephan. Email comm un ication, Ma y 2007. [T ad02] K. T adaki. A generalization of Ch aitin’s halting probabilit y Ω and halt ing self-similar s ets, Hokkaido Math. J. , 31:219–25 3, 2002. [vL90] M. v an Lam balgen. The axiomatiza tion of randomness, The Journal of Symb olic L o gic , 55:114 3–116 7, 1990. 23 [Zam90] D. Zamb ella. O n sequences with simp le initial segmen ts, 1990 . ILLC T ec hn ical Rep ort ML 1990 -05, Unive rsit y of Amsterdam. [Zim07] M. Zimand. Tw o sources are b etter than on e for increasing the Kol- mogoro v complexit y of infinite sequences, 2007. CO RR T ec hical Rep ort. arXiv:0705 .4658 . [ZL70] A. Z v onkin and L. Levin. The complexity of fin ite ob jects and the dev elop- men t of th e concepts of information an d randomness by means of the theory of algorithms, R ussian Mathematic al Surveys , 25(6) :83–12 4, 1970. 24