Lower and Upper Expected Hitting Times for Weighted Imprecise Markov Chains

Lo w er and Upp er Exp ected Hitting Times for W eigh ted Imprecise Mark o v Chains Marco Sangalli and Thomas Krak Eindho ven Univ ersity of T ec hnology , Eindhov en, m.sangalli@tue.nl Abstract. In this paper, we extend hitting times for imprecise Mark ov c hains to the framework of weigh ted imprecise Mark ov c hains (WIMCs), in which eac h transition is asso ciated with a strictly positive weigh t en- co ded b y a matrix 𝑊 . Given a con vex set T of admissible transition matrices, w e define low er and upper exp ected hitting times for WIMCs as the infimum and supremum of the (weigh ted) exp ected hitting times o ver T , and we characterise these quantities as the unique solutions of nonlinear fixed-p oin t equations. W e sho w that any w eighted hitting time problem can b e transformed into an un weigh ted hitting time problem on an augmen ted state space, enabling the reuse of existing IMC theory and algorithms. In particular, w e are able to adapt kno wn iterative metho ds for the numerical computation of exp ected hitting times for WIMCs. Keyw ords: Imprecise Marko v chain, Imprecise Probabilit y , Exp ected Hitting Times, W eighted Marko v chain 1 In tro duction and Motiv ation Hitting times are a fundamen tal quantitativ e measure for sto chastic pro cesses. Giv en a target set of states 𝐴 , the exp ected hitting time measures ho w long a pro cess needs, on av erage, to reac h the target 𝐴 . In classical precise Marko v c hains this notion is usually understo o d as the exp ected num ber of transitions un til first arriv al. A first generalisation replaces step counts with a matrix of nonnegative w eights 𝑊 that assigns a trav el time (or cost) to each transition. The result- ing ob ject is a w eighted Mark ov c hain, and the corresp onding hitting time is defined as the accum ulated weigh t along a random tra jectory up to the first en trance in 𝐴 . This naturally arises in man y applications, ranging from routing and transp ortation to rob otic surv eillance [1, 3]. A second generalisation addresses mo del uncertain ty: instead of a single tran- sition matrix, one considers a set T of possible transition matrices, leading to an imprecise Mark ov chain (IMC) [2, 5]. F or IMCs, low er and upp er hitting times are obtained b y minimising or maximising the exp ected hitting time o ver all admissible Mark ov c hains. These t wo generalisations ha ve so far b een studied largely indep enden tly: w eighted trav el times for precise c hains on the one hand, and (un weigh ted) exp ected hitting times for IMCs on the other. 2 Marco Sangalli et al. In this pap er, we combine these tw o p ersp ectiv es by extending the notion of exp ected hitting times for IMCs to the weigh ted setting. W e introduce and study weighte d impr e cise Markov chains (WIMCs), in which transitions are b oth un- certain, through a nonempty set T of transition matrices, and asso ciated with strictly positive weigh ts giv en b y a matrix 𝑊 . In this context, we in troduce lo wer and upper exp ected hitting times as the infim um and supremum of the (w eighted) hitting time o v er T , and w e characterise these quantities as the unique solutions of nonlinear fixed-p oin t equations. The key insight is that an y w eighted hitting time problem can b e transformed into an unw eighted hitting time prob- lem on an augmen ted state space. This connection enables us to adapt and reuse existing IMC theory and algorithms to WIMCs: we are able to show that there alw ays exist transition matrices attaining the minim um and maxim um in the definition of low er and upp er hitting time for WIMCs, that these quantities are c haracterised as the unique solutions of tw o nonlinear fixed-p oin t equations, and that w e can compute them efficiently adapting Krak’s [6] algorithm for IMCs. The remainder of the paper is organised as follows. Section 2 recalls basic notation and classical results on hitting times and Marko v chains. Section 3 in tro duces weigh ted Mark ov c hains and presents key results connecting the un- w eighted and weigh ted framework. Section 4 defines weigh ted imprecise Marko v c hains, presents the main fixed-p oin t results, and discusses algorithmic asp ects. 2 Preliminaries Let X be a discrete finite space with cardinality 𝑁 ∈ N . A discrete-time stochastic pro cess on X is a sequence of random v ariables ( 𝑋 𝑛 ) 𝑛 ∈ N 0 taking v alues in X . The pro cess ( 𝑋 𝑛 ) 𝑛 ∈ N 0 is said to b e a Markov chain if it satisfies P 𝑋 ( 𝑋 𝑛 + 1 = 𝑥 𝑛 + 1 | 𝑋 0: 𝑛 = 𝑥 0: 𝑛 ) = P 𝑋 ( 𝑋 𝑛 + 1 = 𝑥 𝑛 + 1 | 𝑋 𝑛 = 𝑥 𝑛 ) (1) for all 𝑥 0 , . . . , 𝑥 𝑛 + 1 ∈ X and all 𝑛 ∈ N 0 , and where we write P 𝑋 for the probability measure asso ciated to the pro cess ( 𝑋 𝑛 ) 𝑛 ∈ N 0 . The Marko v c hain ( 𝑋 𝑛 ) 𝑛 ∈ N 0 is said to b e (time-) homo gene ous if P 𝑋 ( 𝑋 𝑛 + 1 = 𝑦 | 𝑋 𝑛 = 𝑥 ) = P 𝑋 ( 𝑋 1 = 𝑦 | 𝑋 0 = 𝑥 ) (2) for all 𝑥 , 𝑦 ∈ X and all 𝑛 ∈ N 0 . The 𝑁 by 𝑁 sto c hastic matrix 𝑇 defined by 𝑇 ( 𝑥 , 𝑦 ) : = P 𝑋 ( 𝑋 1 = 𝑦 | 𝑋 0 = 𝑥 ) is the tr ansition matrix of the homogeneous Marko v c hain ( 𝑋 𝑛 ) 𝑛 ∈ N 0 and uniquely c haracterises its b eha viour up to its initial distribution. One may view a homo- geneous Mark ov c hain as a random walk on the directed graph 𝐺 𝑇 = ( 𝑉 , 𝐸 ) where 𝑉 : = X and ( 𝑥 , 𝑦 ) ∈ 𝐸 if and only if 𝑇 ( 𝑥 , 𝑦 ) > 0. Let 𝐴 ⊂ X b e a nonempty target set of states. The hitting time of the Mark o v c hain ( 𝑋 𝑛 ) 𝑛 ∈ N 0 is the random v ariable defined as 𝜏 𝐴 : = inf { 𝑛 ≥ 0 : 𝑋 𝑛 ∈ 𝐴 } ∈ N 0 ∪ { +∞ } , (3) Exp ected Hitting Times for WIMCs 3 and, for a starting state 𝑥 ∈ X , the exp e cte d hitting time is ℎ 𝑇 ( 𝑥 ) : = E P 𝑋  𝜏 𝐴 | 𝑋 0 = 𝑥  . (4) The quantit y ℎ 𝑇 ( 𝑥 ) can b e seen as the mean num b er of steps the chain ( 𝑋 𝑛 ) 𝑛 ∈ N 0 tak es b efore reaching the target 𝐴 , and is the minimal nonnegativ e solution of a linear system [8]        ℎ 𝑇 ( 𝑥 ) = 0 if 𝑥 ∈ 𝐴 , ℎ 𝑇 ( 𝑥 ) = 1 + Í 𝑦 ∈ X 𝑇 ( 𝑥 , 𝑦 ) ℎ 𝑇 ( 𝑦 ) if 𝑥 ∉ 𝐴 . (5) 3 W eighted Marko v Chains Let 𝑇 b e a transition matrix on X and let 𝑊 > 0 b e a 𝑁 b y 𝑁 matrix of w eights. The quantit y 𝑊 ( 𝑥 , 𝑦 ) can be understoo d as the time or cost that the homogeneous Marko v c hain ( 𝑋 𝑛 ) 𝑛 ∈ N 0 go verned b y 𝑇 tak es to transition from 𝑥 to 𝑦 . W e call the pair ( ( 𝑋 𝑛 ) 𝑛 ∈ N 0 , 𝑊 ) a weighte d Markov chain . W e can naturally define the hitting time for a weigh ted Marko v c hain as [1, 3] 𝜂 𝑊 𝐴 : = 𝜏 𝐴  𝑛 = 1 𝑊 ( 𝑋 𝑛 − 1 , 𝑋 𝑛 ) . (6) Conditioned on the c hain starting in 𝑥 ∈ X , the exp e cte d hitting time for a w eighted Marko v chain is ℎ 𝑇 ,𝑊 ( 𝑥 ) : = E P 𝑋  𝜂 𝑊 𝐴 | 𝑋 0 = 𝑥  . (7) This quantit y represen ts the exp ected trav el time from a state 𝑥 ∈ X to 𝐴 . Similarly as b efore, the exp ected hitting time satisfies a system of equations [1]:        ℎ 𝑇 ,𝑊 ( 𝑥 ) = 0 if 𝑥 ∈ 𝐴 , ℎ 𝑇 ,𝑊 ( 𝑥 ) = Í 𝑦 ∈ X 𝑇 ( 𝑥 , 𝑦 )  𝑊 ( 𝑥 , 𝑦 ) + ℎ 𝑇 ,𝑊 ( 𝑦 )  if 𝑥 ∉ 𝐴 . (8) The next result follo ws directly from the linearity in 𝑊 of ℎ 𝑇 ,𝑊 , as in Eq. (8). Lemma 1. L et 𝑐 > 0 . Then ℎ 𝑇 ,𝑐 𝑊 = 𝑐 ℎ 𝑇 ,𝑊 . The next result builds a bridge b et ween w eighted and unw eigh ted Mark o v chains, i.e, under the condition that 𝑊 ≥ 2, for ev ery weigh ted Mark ov c hain w e can construct a (un weigh ted) Marko v c hain with the same exp ected hitting time. Prop osition 1. L et 𝑇 b e a tr ansition matrix on X , let 𝐴 ⊂ X b e the tar get, and let 𝑊 ≥ 2 . We c an c onstruct a sp ac e X ′ : = 𝑋 ∪ Z with Z : = { ( 𝑥 , 𝑦 ) ∈ X 2 : 𝑇 ( 𝑥 , 𝑦 ) > 0 } and a tr ansition matrix 𝑇 ′ on X ′ such that ℎ 𝑇 ′ ( 𝑥 ) = ℎ 𝑇 ,𝑊 ( 𝑥 ) for al l 𝑥 ∈ X , wher e ℎ 𝑇 ′ and ℎ 𝑇 ,𝑊 ar e the ve ctors of exp e cte d hitting times for the unweighte d (r esp. weighte d) Markov chain on X (r esp. X ′ ) governe d by 𝑇 (r esp. 𝑇 ′ ). 4 Marco Sangalli et al. Pr o of. Let 𝑇 ′ b e the transition matrix on X ′ defined as 1 ) 𝑇 ′ ( 𝑥 , 𝑧 𝑥 𝑦 ) = 𝑇 ( 𝑥 , 𝑦 ) , 2 ) 𝑇 ′ ( 𝑧 𝑥 𝑦 , 𝑦 ) = 1 / ( 𝑊 ( 𝑥 , 𝑦 ) − 1 ) , 3 ) 𝑇 ′ ( 𝑧 𝑥 𝑦 , 𝑧 𝑥 𝑦 ) = 1 − 1 / ( 𝑊 ( 𝑥 , 𝑦 ) − 1 ) , (9) for all 𝑥 , 𝑦 ∈ X and 𝑧 𝑥 𝑦 ∈ Z . Since 𝑊 ≥ 2, 𝑇 ′ is a w ell-defined transition matrix. The follo wing graph represents the situation lo cally . x y x z xy y 𝑇 ( 𝑥 , 𝑦 ) 𝑊 ( 𝑥 , 𝑦 ) 𝑇 ( 𝑥 , 𝑦 ) 1 − 1 𝑊 ( 𝑥 , 𝑦 ) − 1 1 𝑊 ( 𝑥 , 𝑦 ) − 1 F or all 𝑥 ∈ X , the exp ected hitting time of the Mark ov chain on X ′ go verned by 𝑇 ′ satisfies ℎ 𝑇 ′ ( 𝑥 ) = 1 +  𝑧 ′ ∈ X ′ 𝑇 ′ ( 𝑥 , 𝑧 ′ ) ℎ 𝑇 ′ ( 𝑧 ′ ) = 1 +  𝑦 ∈ X 𝑇 ′ ( 𝑥 , 𝑧 𝑥 𝑦 ) ℎ 𝑇 ′ ( 𝑧 𝑥 𝑦 ) . (10) F or all 𝑧 𝑥 𝑦 ∈ Z , the exp ected hitting time satisfies ℎ 𝑇 ′ ( 𝑧 𝑥 𝑦 ) = 1 +  1 − 1 𝑊 ( 𝑥 , 𝑦 ) − 1  ℎ 𝑇 ′ ( 𝑧 𝑥 𝑦 ) +  1 𝑊 ( 𝑥 , 𝑦 ) − 1  ℎ 𝑇 ′ ( 𝑦 ) ⇒ ℎ 𝑇 ′ ( 𝑧 𝑥 𝑦 ) = ℎ 𝑇 ′ ( 𝑦 ) + 𝑊 ( 𝑥 , 𝑦 ) − 1 . (11) W e substitute (11) in to (10) and obtain ℎ 𝑇 ′ ( 𝑥 ) = 1 +  𝑦 ∈ X 𝑇 ′ ( 𝑥 , 𝑧 𝑥 𝑦 ) ℎ 𝑇 ′ ( 𝑧 𝑥 𝑦 ) = 1 +  𝑦 ∈ X 𝑇 ( 𝑥 , 𝑦 )  ℎ 𝑇 ′ ( 𝑦 ) + 𝑊 ( 𝑥 , 𝑦 ) − 1  =  𝑦 ∈ X 𝑇 ( 𝑥 , 𝑦 )  𝑊 ( 𝑥 , 𝑦 ) + ℎ 𝑇 ′ ( 𝑦 )  , whic h is exactly the system in Eq. (8). Then, ℎ 𝑇 ′ ( 𝑥 ) = ℎ 𝑇 ,𝑊 ( 𝑥 ) for all 𝑥 ∈ X . ⊓ ⊔ The follo wing result extends Prop osition 1 to the general case 𝑊 > 0. Corollary 1. L et 𝑇 b e a tr ansition matrix on X , let 𝐴 ⊂ X b e the tar get, and let 𝑊 > 0 . We c an c onstruct a sp ac e X ′ : = 𝑋 ∪ Z and a tr ansition matrix 𝑇 ′ on X ′ as in Pr op osition 1 so that ℎ 𝑇 ,𝑊 ( 𝑥 ) = 1 𝑐 ℎ 𝑇 ′ ( 𝑥 ) for some 𝑐 ≥ 1 and for al l 𝑥 ∈ X , wher e ℎ 𝑇 ′ and ℎ 𝑇 ,𝑊 ar e the ve ctors of exp e cte d hitting times for the unweighte d (r esp. weighte d) Markov chain on X (r esp. X ′ ) governe d by 𝑇 (r esp. 𝑇 ′ ). Exp ected Hitting Times for WIMCs 5 Pr o of. Let 𝑤 𝑚 : = min 𝑥 , 𝑦 ∈ X 𝑊 ( 𝑥 , 𝑦 ) and define 𝑊 ′ : = 𝑐𝑊 , where 𝑐 : = max { 1 , 2 / 𝑤 𝑚 } . It follows that 𝑊 ′ ≥ 2 and, b y Lemma 1, w e ha v e 𝑐 ℎ 𝑇 ,𝑊 = ℎ 𝑇 ,𝑐 𝑊 = ℎ 𝑇 ,𝑊 ′ . By follo wing the construction in the pro of of Prop osition 1, we can find a transition matrix 𝑇 ′ on X ′ suc h that ℎ 𝑇 ′ ( 𝑥 ) = ℎ 𝑇 ,𝑊 ′ ( 𝑥 ) = 𝑐 ℎ 𝑇 ,𝑊 ( 𝑥 ) for all 𝑥 ∈ X , whic h is what w e wan ted to prov e. ⊓ ⊔ 4 W eighted Imprecise Marko v Chains Instead of a single homogeneous Marko v chain ( 𝑋 𝑛 ) 𝑛 ∈ N 0 with a fixed transition matrix 𝑇 , w e now consider a set T of admissible transition matrices. Using this set, we define the family P that con tains all homogeneous Marko v chains whose transition matrix is in T . This set P is called an impr e cise Markov chain (IMC) [2, 5, 4]. Throughout this paper, we impose some conditions on the set T : w e assume that it is nonempt y , compact, con v ex, and has separately sp ecified ro ws (SSR) [5, 7]. This last property means that the set T is the Cartesian pro duct of 𝑁 = | X | compact and conv ex sets of probability distributions { T 𝑥 } 𝑥 ∈ X , one for eac h state 𝑥 ∈ X . Giv en a set of transition matrices T on X , we define the lower and upp er exp e cte d hitting times as follows: ℎ T : = inf 𝑇 ∈ T ℎ 𝑇 and ℎ T : = sup 𝑇 ∈ T ℎ 𝑇 . Krak [6] sho wed that, under the condition (R1): for all 𝑇 ∈ T and all 𝑥 ∈ X there exists 𝑛 ∈ N 0 suc h that [ 𝑇 𝑛 1 𝐴 ] ( 𝑥 ) > 0, i.e that the pro cess can reach 𝐴 with every matrix regardless of the starting state, the v ector of low er hitting times is finite and the unique solution of a system of equations: ℎ T = 1 𝐴 𝑐 + 1 𝐴 𝑐 · 𝑇 ℎ T , (12) where 1 is the indicator function and 𝑇 : R X → R X is the lo wer transition op erator asso ciated to T , given by: [ 𝑇 𝑓 ] ( 𝑥 ) : = inf 𝑇 ∈ T [ 𝑇 𝑓 ] ( 𝑥 ) , for all 𝑓 ∈ R X . (13) A completely analogous c haracterisation also holds for upp er hitting times. Let 𝑊 > 0 b e a matrix of weigh ts. W e define the pair made of an IMC P and a w eight matrix 𝑊 as a weighte d impr e cise Markov chain (WIMC). Given a set of transition matrices T , w e define the low er and upp er exp ected hitting times for a WIMC as ℎ 𝑊 : = inf 𝑇 ∈ T ℎ 𝑇 ,𝑊 and ℎ 𝑊 : = sup 𝑇 ∈ T ℎ 𝑇 ,𝑊 . (14) W e w ant to characterise these quantities as the unique solutions of tw o nonlinear fixed-p oin t equations. W e b egin with the following lemma. 6 Marco Sangalli et al. Lemma 2. L et T b e a set of tr ansition matric es on X . Then, ther e exists 𝑇 , ˜ 𝑇 ∈ T such that ℎ 𝑇 ,𝑊 = ℎ 𝑊 and ℎ ˜ 𝑇 ,𝑊 = ℎ 𝑊 . (15) Pr o of. By Krak et al. [7, Theorem 12], if 𝑊 = 1 there exists a matrix 𝑇 ∈ T that ac hieves the minimum in the definition of lo wer hitting time. Using the construction and notation from Proposition 1 and Corollary 1, w e find that there exists a transition matrix 𝑇 ′ on X ′ = X ∪ Z achieving the lo wer hitting time, thus the transition matrix 𝑇 on X defined as 𝑇 ( 𝑥 , 𝑦 ) = 𝑇 ′ ( 𝑥 , 𝑧 𝑥 𝑦 ) ac hieves the minim um in ℎ 𝑊 . An analogous argumen t holds for upp er hitting times. ⊓ ⊔ Define the op erator ℒ 𝑊 : R X → R X as ( ℒ 𝑊 𝑓 ) ( 𝑥 ) =          0 if 𝑥 ∈ 𝐴 , inf 𝑇 ∈ T " Í 𝑦 ∈ X 𝑇 ( 𝑥 , 𝑦 )  𝑊 ( 𝑥 , 𝑦 ) + 𝑓 ( 𝑦 )  # if 𝑥 ∉ 𝐴 , (16) for all 𝑓 ∈ R X . The following theorem states that ℎ 𝑊 is the unique solution of a nonlinear fixed-p oin t equation. Theorem 1. L et T b e a set of tr ansition matric es on X . Then, under c ondition (R1), ℎ 𝑊 is the unique solution to ℎ 𝑊 = ℒ 𝑊 ℎ 𝑊 . (17) Pr o of. If 𝑊 ≥ 2, w e follow the construction and notation of Prop osition 1 and w e obtain an (unw eighted) imprecise Marko v chain on X ′ = X ∪ Z with set of transition matrices T ′ . W e know from Krak [6, Prop osition 2] that there exists a unique ℎ T ′ ∈ R X ′ that satisfies ℎ T ′ ( 𝑥 ) = ( 0 if 𝑥 ∈ 𝐴 , 1 + inf 𝑇 ′ ∈ T ′ [ 𝑇 ′ ℎ T ′ ] ( 𝑥 ) if 𝑥 ∉ 𝐴 . In particular, there exists a matrix ˆ 𝑇 ′ ∈ T ′ satisfying ℎ ˆ 𝑇 ′ = ℎ T ′ . The transi- tion matrix ˆ 𝑇 ∈ T defined as ˆ 𝑇 ( 𝑥 , 𝑦 ) = ˆ 𝑇 ′ ( 𝑥 , 𝑧 𝑥 𝑦 ) for all 𝑥 , 𝑦 ∈ X ac hieves the minim um: ℎ ˆ 𝑇 ,𝑊 ( 𝑥 ) = ℎ ˆ 𝑇 ′ ( 𝑥 ) = ℎ T ′ ( 𝑥 ) = inf 𝑇 ′ ∈ T ′ ℎ 𝑇 ′ ( 𝑥 ) = inf 𝑇 ∈ T ℎ 𝑇 ,𝑊 ( 𝑥 ) = ℎ 𝑊 ( 𝑥 ) , for all 𝑥 ∈ X . It follows that ℎ 𝑊 ( 𝑥 ) = ℎ ˆ 𝑇 ,𝑊 ( 𝑥 ) = ℎ ˆ 𝑇 ′ ( 𝑥 ) = ℎ T ′ ( 𝑥 ) = 1 + inf 𝑇 ′ ∈ T ′ [ 𝑇 ′ ℎ T ′ ] ( 𝑥 ) = 1 + inf 𝑇 ′ ∈ T ′  𝑦 ∈ X 𝑇 ′ ( 𝑥 , 𝑧 𝑥 𝑦 ) ℎ T ′ ( 𝑧 𝑥 𝑦 ) = 1 + inf 𝑇 ′ ∈ T ′  𝑦 ∈ X 𝑇 ′ ( 𝑥 , 𝑧 𝑥 𝑦 )  ℎ T ′ ( 𝑦 ) + 𝑊 ( 𝑥 , 𝑦 ) − 1  = inf 𝑇 ∈ T  𝑦 ∈ X 𝑇 ( 𝑥 , 𝑦 )  ℎ 𝑊 ( 𝑦 ) + 𝑊 ( 𝑥 , 𝑦 )  =  ℒ 𝑊 ℎ 𝑊  ( 𝑥 ) , Exp ected Hitting Times for WIMCs 7 for all 𝑥 ∉ 𝐴 . T rivially , the equality ℎ 𝑊 ( 𝑥 ) = [ ℒ 𝑊 ℎ 𝑊 ] ( 𝑥 ) also holds for all 𝑥 ∈ 𝐴 . W e note that ℎ T ′ ( 𝑧 𝑥 𝑦 ) = ℎ T ′ ( 𝑦 ) + 𝑊 ( 𝑥 , 𝑦 ) − 1 follows directly from (11) in the pro of of Prop osition 1. On the other hand, if 0 < 𝑊 ( 𝑥 , 𝑦 ) < 2 for some 𝑥 , 𝑦 ∈ X we rescale all the w eights with a constant 𝑐 ≥ 1 as in Corollary 1. W e obtain 𝑐 ℎ 𝑊 ( 𝑥 ) = ℎ 𝑐𝑊 ( 𝑥 ) =  ℒ 𝑐𝑊 ℎ 𝑐𝑊  ( 𝑥 ) = h ℒ 𝑐𝑊  𝑐 ℎ 𝑊  i ( 𝑥 ) = inf 𝑇 ∈ T  𝑦 ∈ T 𝑇 ( 𝑥 , 𝑦 )  𝑐𝑊 ( 𝑥 , 𝑦 ) + 𝑐 ℎ 𝑊 ( 𝑦 )  = 𝑐 inf 𝑇 ∈ T  𝑦 ∈ T 𝑇 ( 𝑥 , 𝑦 )  𝑊 ( 𝑥 , 𝑦 ) + ℎ 𝑊 ( 𝑦 )  = 𝑐  ℒ 𝑊 ℎ 𝑊  ( 𝑥 ) , for all 𝑥 ∉ 𝐴 , while equality is trivial for all 𝑥 ∈ 𝐴 . ⊓ ⊔ By defining ℒ 𝑊 taking the supremum in place of the infim um in (16), one can sho w that the upper hitting time ℎ 𝑊 is the unique solution of a fixed-p oint equation: ℎ 𝑊 = ℒ 𝑊 ℎ 𝑊 . (18) 4.1 Computing Low er and Upper Hitting Times Krak [6] prop oses efficient iterative algorithms to compute lo wer and upper hit- ting times for (unw eighted) imprecise Mark o v chains. Thanks to Prop osition 1 and Corollary 1, we can use the same algorithms to compute lo w er and upp er hitting times for weigh ted imprecise Marko v chains. In particular, the algorithm for lo w er hitting times alternates b et ween computing the (weigh ted) hitting time ℎ 𝑇 𝑛 b y solving the linear system in (8) for a matrix 𝑇 𝑛 ∈ T , and improving the transition matrix b y solving 𝑇 𝑛 + 1 ( 𝑥 , ·) = argmin 𝑇 ( 𝑥 , · ) ∈ T 𝑥  𝑦 ∈ X 𝑇 ( 𝑥 , 𝑦 )  𝑊 ( 𝑥 , 𝑦 ) + ℎ 𝑇 𝑛 ( 𝑦 )  , for all 𝑥 ∈ 𝐴 𝑐 . This matrix is identical to the one obtained by constructing the space X ′ = X ∪ Z and the matrix 𝑇 ′ 𝑛 as in Corollary 1, by computing 𝑇 ′ 𝑛 + 1 ∈ T ′ minimising Í 𝑦 ∈ X 𝑇 ′ ( 𝑥 , 𝑧 𝑥 𝑦 ) ℎ 𝑇 ′ 𝑛 ( 𝑧 𝑥 𝑦 ) for all 𝑥 ∈ X as in Krak [6], and finally b y setting 𝑇 𝑛 + 1 ( 𝑥 , 𝑦 ) = 𝑇 ′ 𝑛 + 1 ( 𝑥 , 𝑧 𝑥 𝑦 ) . Then, the conv ergence of this algorithm (and the analogous v arian t for upp er hitting times) is guaranteed b y Krak [6]. Ac knowledgemen ts This work has b een partly supp orted by the P ersOn pro ject (P21-03), which has receiv ed funding from Nederlandse Organisatie v o or W etensc happ elijk Onderzo ek (NWO). Bibliograph y [1] Carron A, P atel R, Bullo F (2016) Hitting time for doubly-w eighted graphs with application to robotic surv eillance. In: Proceedings of the 2016 Europ ean Control Conference (ECC), Aalb org, Denmark, pp 661–665, doi:10.1109/ECC.2016.7810364 [2] de Cooman G, Hermans F, Quaegheb eur E (2009) Imprecise Marko v chains and their limit behavior. Probability in the Engineering and Informational Sciences 23(4):597–635, doi:10.1017/S0269964809990039 [3] Duan X, Bullo F (2021) Mark o v chain–based stochastic strategies for robotic surv eillance. Annual Review of Control, Rob otics, and Autonomous Systems 4:243–264, doi:10.1146/ann urev-control-071520-120123 [4] Hartfiel DJ, Seneta EB (1994) On the theory of Marko v set-c hains. Adv ances in Applied Probabilit y 26(4):947–964, doi:10.2307/1427899 [5] Hermans F, ˇ Skulj D (2014) Sto chastic processes. In: Augustin T, Co olen FP , de Co oman G, T roffaes MC (eds) Introduction to Imprecise Probabilities, Wiley , c hap 11, pp 258–278, doi:10.1002/9781118763117.ch11 [6] Krak T (2021) Computing exp ected hitting times for imprecise Marko v c hains. Space T echnology Pro ceedings 8:185–205, doi:10.1007/978-3-030- 80542-5 12 [7] Krak T, T’Joens N, De Bock J (2019) Hitting Times and Probabilities for Im- precise Mark o v Chains. In: Pro ceedings of the 14th In ternational Symposium on Imprecise Probabilities: Theories and Applications (ISIPT A), Pro ceedings of Mac hine Learning Research, vol 103, pp 265–275, 1905.08781 [8] Norris JR (1997) Marko v Chains. Cam bridge Series in Statistical and Probabilistic Mathematics, Cambridge Univ ersity Press, Cambridge, UK, doi:10.1017/CBO9780511810633