Cellular Offloading via Downlink Cache Placement

1 Cellular Of floading v ia Do wnlink Cache Placement Bojie Lv , Lexiang Huang a nd Rui W ang The Southern Univ ersity of Scie n ece and T echnolog y Email: { lvbj, huanglx } @mail.sus tc.edu.cn , wang.r@sustc.edu .cn Abstract —In this paper , the downlink file transmission withi n a finite lifetime is optimized with t he assistance of wireless cache nodes. Specifically , th e n umber of requests within the lifetime of one fi l e is modeled as a Poisson p oint process. Th e base station multicasts files to downlink users and the selected the cac he nodes, so that the cache nodes can help to forward t he fil es in the next file request. Thus we formulate the downlink transmission as a Marko v d ecision p rocess with random number of stages, where transmission power and time on each transmission are the control policy . Due to random number of file t ransmission s, we first proposed a revised Bellman’s equ ation, where the optimal control policy can be d eriv ed. In order to address the prohibitively huge state sp ace, we also introduce a low-complexity su b-optimal solution based on an linear approximation of th e value functi on. The approximated value function can b e calculated analytically , so that con ventional numerical value iteration can b e eliminated. Moreo ver , the gap between the approximated va lue functi on and the real value function is bound ed analytically . It is shown by simulation that, with the approximated MDP approach, the proposed algorithm can significantly reduce the resourc e consumption at the base station. I . I N T R O D U C T I O N Caching is a p romising te c h nology for fu ture cellular net- works, which could improve th e network sp ectral efficiency [1] or cut down e nergy con sumption [2], [3] by exploiting th e repeated transmissions of th e same co ntent. With the wired connectio n between cache nodes and b a se station (BS), the buf fer capacity limitation at the cache nodes beco mes th e major perf ormance b ottleneck, an d there h av e been a n umber of research efforts spent on the file place m ent of the cache nodes. For example, in order to o ffload more work to cache nodes, a file p lacement a lg orithm was d erived in [4]. A lon g- term file placemen t policy was prop o sed in [3] to m inimize the tr ansmit power in BS. In [ 5], it is shown that pred iction of future dem and will imp rove the p erform ance of file placem ent at th e cach e nod es. All th e a b ove work s assume that there are wired links between the cache nodes and BSs. It might be co stly to deploy cache n odes with wired co nnection in some areas, and hence the cache placemen t v ia wire less links (e.g ., cellular downlink) should also be studied [6]. In [7], the transmissions fr om BS to cache n odes an d fro m cache nodes to users share th e sam e spectrum. Howe ver , it neglects the possibility that both c a che nodes and users can listen to the BS simultaneously via a multicast mode. In fact, if file caching has to b e made via downlink, the phase of cache placemen t can b e co upled with the p hase o f serving requesting users. For example, at the first few transm ission s of one file, both cache nod es and r e questing users can listen to the BS simultaneously; and the cache nodes can help to fo rward the file as lo ng as they have been a b le to deco de it. Thus the downlink strategy sh ould b e optim iz e d spanning th e who le lifetime of a file (includes the transmission to both cache no des and u sers). Howe ver, this h as n ot been addressed by the existing literatu re. In this pap er , we would like to shed some lig ht on the above open issue by c o nsidering the downlink file tran smission with the assis tance of cache nodes, wh ich ca n on ly r eceiv e data via downlink transmission. Th e comm unication links between cache nodes and users ar e via different spectrum from the d ownlink (e.g., W i- Fi) as [8], [9]. He n ce the BS try to minimize the average downlink resou rce con sumption by offloading some o f the traffic to cache no des. Specifically , we mod el the downlink transmission of one file with in his lifetime as a Markov decision pro cess ( MDP) with random number of stages. Note that this is n ot conv entional MDP problem with finite and fixed nu mber of stages, we pr opose a revised version of Bellman’ s equation, where the optimal control p o licy c a n be ob tained given th e value fun ction. Then we intro d uce a linear appr oximation o n the value function so that the exponential complexity can b e redu ced into linear . The bound s on the approx imation error are also obtained. It is shown by simulation s that th e pro posed scheme can significantly suppress the overall transmission consump tion compare d with baselines. The remain d er of this paper is organized as follows. In Section II, the system m odel is in troduced . In Section III, we formu late the downlink resou rce allocation as a MDP with random number of stages. In Section IV, th e ap proach to obtain the optima l contr ol policy is explained. In Section V, a lin ear app roximation is pr oposed to the value function , and the bo und on appr oximation error is d eriv ed. The n umerical simulation is provided in Section VI and the co nclusion is drawn in Section VI I. I I . S Y S T E M M O D E L A. Downlink F ile Requ e st Model The downlink transmission in a cell with one multi-an te n na BS, N C single-anten na ca c h e nodes is consider ed. Let N T be the nu m ber of antennas at the BS. W ithout lo ss of gen erality , it is assumed that the BS locates in the orig in, an d the location s of cache nodes are d enoted as c 1 , c 2 , ..., and c N C respectively , which can be arbitrary in the cell coverage. Let C i ( ∀ i = 1 , 2 , ..., N C ) be the coverage region of the i -th cache n o de. The d ownlink data is o rganized b y files. Similar to most of the existing literature, it is assumed that each file consists o f R f informa tio n b its, and it is further divided into N S segment 2 equally . Th ese files can b e p o pular video s, breakin g news o r other we b con te n t, which might be req uested by multiple users in the cell. No te th at the popular ity of web conten t depends heavily on the time. For example, break ing news may not be popular anymore after one day . T o mo del the dynam ics in file p opularity , it is assumed th at each file h as a lifetime T . One file will be downloaded in the c a che n odes only during its lifetime. For the elabor ation co n venience, w e co nsider the same file size an d lifetime. In fact, our app roach is app licable for h eterogen e ous file sizes a n d lifetimes. Users appea r random ly in the cell coverage to req uest files via BS. Suppose the f -th file ( f = 1 , 2 , ... ) is av ailab le f or access since time instance t f , we conside r th e req uests on th is file happ en d uring the time period [ t f , t f + T ] . The Poisson point pro cess is adopted to mo del the event of file request within a file’ s life time, an d λ f is the p rocess intensity o f the each file. Note that in q ueueing th eory , Poisson poin t p rocess is widely accepted to mo del random arriv al e vents, such as the arriv al of customers a t a store or p h one calls at an exchange. Hence, th e prob ability mass fu nction of request nu mber for the i -th file ( ∀ i ), wh en th e rem aining life time is T r em , is giv en by Pr( Request nu m ber = n ) = ( λ f T r em ) n n ! e − λ f T rem . (1) B. W ir eless Cachin g Model W e consider the scenario that wired conn ections between BS and ca che no des are no t av ailable, h ence the ca c h e nod es can only r eceiv e the files fro m the d ownlink. Comp ared with wired connectio n or dedicated spectrum f or the comm unications between cache n odes and BS, this scen ario provides more flexibility on deploymen t. A s a result, th ere ar e two modes of da ta tran smission in the network. • Downlink multicast: the BS delivers the requested file segments to user and some of the cache nod es. This will happen when th e req u ested segments cannot b e fo und in the cache nodes nearby . For example, if on e user is within the coverage of C i , it will r eceiv e the file segments fro m th e BS when the se file segments can n ot be found in the i - th cache node. Since b o th r equesting user an d ca c he nod es can listen when the BS is tran smitting, the tra nsmission m ode is multicast. • Device-to-device (D2D) communicat io ns: the cache nodes forward the r e q uested file segments to users. This will h a ppen when the req uested segments hit the buffer of nearby cache no des. For e xample, if o ne u ser is within the coverage of C i , it will receive the file segments fr om the i -th cache node when the file segmen ts can be f o und th ere. This commun ication is m ade directly fr om cache no des to user s. It can use Wi-Fi, blu etooth, or other air interfaces, which is not in th e same spectrum as d ownlink. For example, the offloading from cellular to W i-Fi has attra cted a numb er o f attenuation recently [8] [9] . Since the secon d mode of com munication s can be do ne distributi vely and paren tally with relatively low tra n smission power , we consider th e transmission re so urce consum ption at the BS on ly , which is the bo ttleneck of the overall system. The first transmission of each file is m ade directly f r om the BS, while some c ache no d es ma y a lso be able to decode the who le file o r some segmen ts. Hen ce, fo r the fo llowing requests o f the same file, user will receive one segmen t from the cach e node if the fo llowing two condition s are satisfied: (1) the user is in the coverage of certain cach e node ; (2 ) the aforemen tioned file segment has b een successfully decoded b y the aforemen tioned cache nod e. For elaboration convenience, we shall refer to the user, which sends the n -th re q uest o n the f -th file, as the ( f , n ) -th user , and refer to the s -th segment of the f -th file as the ( f , s ) -th segment. Since the data r a te of wireless comm u nications is usually mu ch smaller tha n w ir ed, and each file has finite lifetime, we ign ore the limitation o n cache buffer in this pap er 1 . C. Downlink Ph ysical Layer Model In downlink, the receivers includ e the requesting user and cache nodes, and the space-time block code (STBC) with full d iversity is u sed at the BS to facilitate the multicast commun ications. The ben efits of STBC in multicast are that (1) the BS need not to co llect chann el state inform a tion (CSI); (2) th e full diversity can be achieved at a ll the rece ivers. Since the transmission time of one file segment is much larger th an the chann el co herent time, it is assumed tha t the ergodic chann el capacity span all possible sma ll- scale chan nel fading can be achieved durin g on e segment tra nsmission. Let ρ f ,n and ρ c be the pathloss f rom the BS to the ( f , n ) -th user and th e c - th ca che no de respec tively , η f ,n,s and η c f ,n,s be the correspo n ding shadowing attenuatio n in n -th transmission of the ( f , s ) -th segment, P f ,n,s be the downlink transmission power o f the s -th file segment in response to req uest of the ( f , n ) -th user, N f ,n,s be the numb er of downlink transmission symbols scheduled to deliver the s -th segment to the ( f , n ) -th user , th e thr oughp ut achieved by the ( f , n ) -th d ownlink user in the transm ission of the s -th segment is g i ven by R f ,n,s = N f ,n,s E h f,n,s  log 2  1 + || h f ,n,s || 2 P f ,n,s N T σ 2 z  , (2) where σ 2 z is the power of noise as well as in ter-cell in- terference , h f ,n,s is the i.i.d . (in depend ently an d id entically distributed) channel vector fro m th e BS to the requ est u ser . Each element of h f ,n,s is com p lex Gaussian distributed with zero mean an d variance ρ f ,n η f ,n,s . A s a r emark note that the transmission o f on e segment m ay consume a large numb er of frames, and the ch a nnel vector h f ,n,s can be different from frame to fram e. Howe ver , since we con sider the ergodic chan - nel capacity , the randomn e ss in small- scale fading is av eraged. Hence, th e ( f , n ) -th user can decode the s -th segment only when R f ,n,s ≥ R f / N S . (3) 1 For example , suppose that one BS is transmitting downlin k files with ov erall data rate of 1 Gbps, and the life time of eac h file is 24 hours. Then the maximum required storage capaci ty of one cache node is around 10 T bytes, which is a mild requiremen t for ignoring the buf fer capacit y limitati on. 3 Moreover , the achiev able data rate of th e c -th cache node is given by R c f ,n,s = N f ,n,s E h f,n,s " log 2 1 + || h c f ,n,s || 2 P f ,n,s N T σ 2 z !# , (4) where h c f ,n,s is the i.i.d. channel vector from the BS to c - th cache n ode. Each elem ent of h c f ,n,s is complex Gaussian distributed with zero m ean and variance ρ c η c f ,n,s . The c -th cache nod e can decode the s - th segment only when R c f ,n,s ≥ R f / N S . In the downlink transmission of o ne file, the loca tion of requesting user , thus ρ f ,n , is assumed to be static. Mor eover , the shadowing effect η f ,n,s and η c f ,n,s is assumed to be i. i. d. for d ifferent segments. D. Contr ol P o licy There is an scheduling issue in the downlink tran smission: if mo r e power or time resource is sp ent in downlink file transmission, more cache node s are able to buffer the file (or some segments of the file), which may sav e th e resource o f th e BS in th e successive transmission of the same file. Thu s it is necessary to optimize the reso u rce allocation during the whole lifetime of on e file, instead of single transmission. Since this is a multi-stag e optimization pro blem, we shall fo rmulate it as finite-stage MD P . In this section, the system state and co ntrol policy will be d efined. Definition 1 (System State) . Befor e the transmission to the ( f , n ) -th user , the system status is u niquely specified b y S f ,n = h B c f ,s , ρ f ,n , η f ,n,s , η c f ,n,s |∀ c = 1 , ..., N C ; s = 1 , ..., N S i , wher e B c f ,s = 1 mean s that the s - th se gment of the f -th file has be e n successfully decode d by the c -th ca che node a nd B c f ,s = 0 means otherwise. Then the downlink sched uling policy is d efined as follows. Definition 2 (Control Policy) . Suppose the s -th se gment is transmitted to th e ( f , n ) -th u ser via downlink. Given the system state S f ,n , th e scheduling p olicy Ω f ,n ( ∀ f , n ) is a mapping fr om system state and the r emaining lifetime T f ,n to the schedulin g parameters P f ,n,s and N f ,n,s ( ∀ s ). Thus Ω f ,n ( S f ,n , T f ,n ) = { ( P f ,n,s , N f ,n,s ) |∀ s } . Mo r eover , in order to g uarantee the r eq uesting u ser c an successfully decod e th e downlink data, the constraint in (3) should be satisfied . I I I . P RO B L E M F O R M U L AT I O N In this section, we sha ll formulate the downlink resour ce control prob lem a s a MDP with rand o m n umber of stages. Let C s f ,n = ∪ ∀ i, B i f,s =1 C i be the area whe re the r equesting users is able to rece i ve the s -th segment of the f -th file from one of the cache nod es, and l f ,n be the lo cation of th e ( f , n ) -th user . Hence we use the fo llowing cost func tio n to measure the weighted sum of energy an d tran smission time of the BS, which is spent on the ( f , n ) -th user . g f ,n,s = I ( l f ,n / ∈ C s f ,n ) × ( w e P f ,n,s N f ,n,s + w t N f ,n,s ) , where w e and w t are the weights on transmission energy and transmission o pportu n ities respectively , and I ( · ) is the indication fu nction. Since transmission f r om cache n odes to users is mad e v ia other air inter faces, the transmission resource used by th e cach e n odes is no t co unted in this co st function . Hence th e average cost spent on the f -th file is given by g f ( { Ω f ,n |∀ n } ) = X N E η, ρ " ( λ f T ) N N ! e − λ f T N X n =1 N S X s =1 g f ,n,s # , where the expectation is taken over all possible large-scale channel fading (inclu ding the shadowing effect η an d p athloss ρ ) in th e system . The summ ation on N is due to the r andom number of req uests as elaborated in ( 1). As a result, the overall system cost function is giv en by G ( { Ω f ,n |∀ f , n } ) = lim F → + ∞ 1 F F X f =1 g f ( { Ω f ,n |∀ n } ) , and th e system optimizatio n problem can be written as Problem 1 (Overall System Optimizatio n) . min { Ω f,n |∀ f ,n } G ( { Ω f ,n |∀ f , n } ) s.t. (3) , ∀ f , n, s. Since there is no con straint o n the cache nodes’ buffer capacity and tr a n smission r esources, the above op tim ization problem can b e further deco upled into the f ollowing sub- problem s with respec t to each file. Problem 2 (Optim ization on the f -th File) . min { Ω f,n |∀ n } g f ( { Ω f ,n |∀ n } ) s.t. (3) , ∀ n, s. I V . O P T I M A L C O N T R O L P O L I C Y Note that Problem 2 is a d ynamic programming problem with r andom numb e r o f stage s, which cannot be solved by the standard approach as [10]. W e shall shown in this section that the o ptimal solutio n of Pr oblem 2 (i.e. policy iteration) can be obtained by a p plying value iteration on another MDP pr oblem (Problem 3 as follows) with fin ite an d fixed num ber of stages first, th en solving a revised version o f Bellman’ s eq uation. First of all, we intr oduce the following MDP prob lem with fixed numb er of stage s. Problem 3 (Optim ization with Fixed Stage Number) . min { Ω f,n |∀ n } E η, ρ [ N R X n =1 N S X s =1 g f ,n,s ] s.t. (3) , ∀ n, s. wher e N R is the fixe d numb er o f requests on the f -th file. As introduced in [10], there is standard solution for the MDP pro blem with fin ite and fixed number of stages. The optimal solution of Problem 3 can be deduc e d v ia the Bellman’ s eq uation in (5) on the top of next pag e , w h ere V N R − n +1 ( S f ,n ) is u su ally n amed as value function of the n - th stage, an d S f ,n +1 denotes the next state o f the f -th file. In fact, V N R − n +1 ( S f ,n ) mean s the a verage rem aining cost of the 4 V N R − n +1 ( S f ,n ) = min Ω f,n ( S f,n )  X s g f ,n,s ( S f ,n , Ω f ,n ) + X S f,n + 1 V N R − n ( S f ,n +1 ) P r ( S f ,n +1 | S f ,n , Ω f ,n )  (5) e V N R − n +1 ( e S f ,n ) = min Ω f,n ( e S f,n ) E η, ρ  X s g f ,n,s ( S f ,n , Ω f ,n ) + X e S f,n + 1 e V N R − n ( e S f ,n +1 ) P r ( e S f ,n +1 | S f ,n , Ω f ,n )  (6) Ω ∗ f ,n ( e S f ,n ,T f ,n ) = arg min Ω f,n ( e S f,n ) E η, ρ  X s g f ,n,s ( S f ,n , Ω f ,n ) + X e S f,n + 1 ,N ( λ f T f ,n ) N N ! e − λ f T f,n e V N ( e S f ,n +1 ) P r ( e S f ,n +1 | S f ,n , Ω f ,n )  (7) e V N R − n +1 ( e S f ,n ) ≈ e V N R − n +1 ( e S ∗ f ) + X { ( i,s ) |∀B i f,s ( e S f,n )=0 }  e V N R − n +1 ( e S i,s f ) − e V N R − n +1 ( e S ∗ f )  (8) f -th file from the n -th tran smission to the N R -th transmission, giv en the system state of the n -th stage S f ,n . Note that the large- scale fading is i.i.d. in e a ch file tr ans- mission, the expec tatio n on large-scale fading ca n be taken o n both side of the above equatio n . Hence we h ave the following conclusion , whose p roof is straigh tforward and neglected h ere. Lemma 1 (Bellman’ s Equation with Reduced Space) . The optimal contr ol policy of Pr o blem 3 is th e solution of the Be ll- man’s equation with r ed u ced state space in (6), wher e e S f ,n = {B c f ,s ∈ S f ,n |∀ c, s } , e V N R − n ( e S f ,n ) = E η, ρ [ V N R − n ( S f ,n +1 )] , and Ω f ,n ( e S f ,n ) = { Ω f ,n ( S f ,n ) |∀ ρ f ,n , η f ,n,s , η c f ,n,s , c } . The standard value iteration can be used to solve the Bellman’ s equation (6), and o b tain the value function e V N R − n +1 ( e S f ,n ) ( ∀ n ). I n th e following lemm a, we show that the optimizatio n prob lem o f this paper (Prob lem 2) can also be solved via the above value func tion. Lemma 2 (Optimal Contr ol Po licy of Pr oblem 1) . W ith the value fun ction e V N R − n +1 ( e S f ,n ) ( ∀ n ), the optimal contr ol policy for Pr o blem 2, denoted as Ω ∗ f ,n ( e S f ,n , T f ,n ) , can be calculated fr om (7). Pr oo f. Equation ( 7) is straigh tforward due to the factor that (1) e V N ( e S f ,n +1 ) deno te s the averaged c ost for N time s of requests on the f -th file, give the system state e S f ,n +1 ; (2) ( λ f T f,n ) N N ! e − λ f T f,n is th e p r obability that there are N times of file re q uests within the duration T f ,n . As a result, the optimal co ntrol p olicy of the Problem 2 can be solved via the fo llowing two step s. • V alue Iteration : Calculate the v alue function e V N R − n +1 ( e S f ,n )( ∀ n, e S f ,n ) via B ellman’ s equation with reduced space (6). • Policy Itera tion : Obtain optima l control p olicy from (7). Note th at the state space of the system is ac tually hu g e. For example, suppo se there are 20 cach e nod es in the system and 50 segments per file, the dim e nsional of sy stem state is 2 1000 , which is refer red to as the curse of dimen sionality . Th e o ptimal solution is actually com putationally in tractable. He n ce, we continue to prop ose a low-complexity solution based on th e technique o f app roximated MDP in the fo llowing section. V . L O W - C O M P L E X I T Y S O L U T I O N V I A A P P R OX I M A T E D M D P In this section , we shall first introd uce a novel lin ear approx imation on the value fu nction e V N R − n +1 ( e S f ,n ) , and elaborate the on-line c o ntrol policy to determ ine the co n trol actions giv en the cu rrent system state and approx im ated value function . A. Appr oximation on V alue Function W e first de fine the notation s for following referenc e system states. • e S ∗ f is the state of f -th file where all the cache n odes have succ essfully decoded the whole file. Thus e S ∗ f ,n = {B c f ,s = 1 |∀ c = 1 , ..., N C } . • e S i,s f is the state o f f -th file transmission where on ly the s -th segment at the i -th cache n o de is no t succ e ssfully decoded . e S i,s f = {B i f ,s = 0 , B j f ,t = 1 |∀ ( j, t ) 6 = ( i, s ) } . Hence, we approx imate the value func tio n e V N R − n +1 ( e S f ,n ) as (8), wh ere B i f ,s ( e S f ,n ) means the parame te r of B i f ,s in the system state e S f ,n . In order to app ly this ap proxim a tio n on all value f unction, it is necessary to obtain the value of e V N R − n +1 ( e S ∗ f ) an d e V N R − n +1 ( e S i,s f ) fo r all n, i , and s v ia (6). Generally , th ey can be ev alu ated via nu merically simu lation. In the following, howe ver , we provide the analytically expressions for them. 1) Evalua tion of e V N R − n +1 ( e S ∗ f ) : Note that system state e S ∗ f represents the situation th at all th e cache no des have alr eady decoded th e f -th file, the p urpose o f d ownlink transmission is only to make sure that the requ e sting u sers, which are outside of the c overage of cache n odes, can de c ode the downlink file. Hence it is clear that e V N R − n +1 ( e S ∗ f ) = ( N R − n + 1 ) Pr( l f ,n / ∈ C s f ,n ) E ρ,η  min [ P f,n,s N f,n,s ] w e P f ,n,s N f ,n,s + w t N f ,n,s | l f ,n / ∈ C s f ,n  s.t. R f ,n,s ≥ R f / N S , ∀ s. The ab ove value fun c tion can b e calculated with analytical expression, wh ich is elab o rated below . 5 Lemma 3. Th e va lue function e V N R − n +1 ( e S ∗ f ) is g ive n by e V N R − n +1 ( e S ∗ f ) ≈ ( N R − n + 1 ) Pr( l f ,n / ∈ C s f ,n ) E ρ,η  X s w e P ∗ f ,n,s N ∗ r,f,s + w t N ∗ f ,n,s | l f ,n / ∈ C s f ,n  , wher e P ∗ f ,n,s = w t w e W ( 2 θ w t ew e ) , N ∗ f ,n,s = R f N S [ θ +log 2 ( P ∗ f,n,s )] , θ = E h f,n,s h log 2  || h f,n,s || 2 N T σ 2 z i , a nd W ( x ) is the Lambert-W function [11 ]. Pr oo f. Please ref e r to Ap pendix A. 2) Evalua tion of e V N R − n +1 ( e S i,s f ) : Giv en system state e S i,s f for arbitrary stage, there are only two possible next system states e S i,s f and e S ∗ f , wh ich are discussed b elow . • Wh e n ρ f ,n η f ,n,s ≤ ρ i η i f ,n,s , th u s R f ,n,s ≤ R i f ,n,s , the i -th ca che node is alway able to decode th e s -th file segment giv e that the transmission constraint (3) should be satisfied. Thus the next state must be e S ∗ f . In th is case, the o ptimized RHS of (6) is given by Q c = E min Ω f,n ( S f,n )  X s g f ,n,s ( S f ,n , Ω f ,n )  + e V N R − n ( e S ∗ f ) , subject to con straint (3). • When ρ f ,n η f ,n,s > ρ i η i f ,n,s , thus R f ,n,s > R i f ,n,s , the BS can cho ose to secure th e transmission o f the s - th segmen t to the ( f , n ) - th user or the i -th c a che node. Hence the op timized RHS of (6) is given by E min  Q u ( S f ,n ) , Q i,s ( S f ,n )  , wh ere Q u and Q i,s are defined in (9) and (10) r espectiv ely . As a result, the expr ession of e V N R − n +1 ( e S i,s f ) is sum m arized by th e fo llowing lemma. Lemma 4. The value function e V N R − n +1 ( e S i,s f ) is g iven by (11). Mor eover , the optimal con tr ol actions for Q c and Q u ar e the same as Lemma 3. The op timal co ntr ol ac- tion fo r Q i,s is g ive n by P f ,n,s = w t w e W ( 2 θ i w t ew e ) , N f ,n,s = R f N S [ θ i +log 2 ( P f,n,s )] , θ i = E h i f,n,s h log 2  || h i f,n,s || 2 N T σ 2 z i ; and ∀ t 6 = s , P ∗ f ,n,t = w t w e W ( 2 θ w t ew e ) , N ∗ f ,n,t = R f N S [ θ +log 2 ( P ∗ f,n,t )] , θ = E h f,n,t h log 2  || h f,n,t || 2 N T σ 2 z i . Pr oo f. The proof is similar to that of Lemma 3, and it is omitted h ere. B. Online Co n tr ol W ith the value function e V N R − n +1 ( e S ∗ f ) and e V N R − n +1 ( e S i,s f ) , the value function for arbitrar y system state in arbitrary transmission stag e can be ap proxim ated v ia (8) . Hence th e online contr ol a c tion for arbitrary system state S f ,n , deno ted as Ω ∗ f ,n ( S f ,n ) , can be obtained the following optimizatio n problem . Problem 4 (Onlin e Optimization ) . Ω ∗ f ,n ( S f ,n ) = arg min X s g f ,n,s ( S f ,n , Ω f ,n ) + X N ( λ f T f ,n ) N N ! e − λ f T f,n e V N ( e S f ,n +1 ) s.t. (3) , ∀ s. Since the v alue f unction e V N ( e S f ,n +1 ) is approximated b y (8), the op timization in Problem 4 can be further d ecoupled for each segment. For the s - th segment ( ∀ s ), the solution of Problem 4 can be obtained b y the fo llowing prob lem, given that the requesting user canno t find the segment f rom n earby cache n odes. Problem 5 (Onlin e Optimization for th e s - th Segment) . { P ∗ f ,n,s , N ∗ f ,n,s } = arg min g f ,n,s ( S f ,n , Ω f ,n ) + X N ( λ f T f ,n ) N N ! e − λ f T f,n X { i |∀B i f,s ( e S f,n + 1 )=0 } e V N ( e S i,s f ) − e V N ( e S ∗ f ) s.t. (3) , wher e e S f ,n +1 is the n ext system state, and B i f ,s ( e S f ,n +1 ) r ep r e sents the buffer sta tus for the ( f , s ) -th se gmen t in the i -th ca che n ode. Due to the secon d term of objec tive in Pr oblem 5, the BS should fir st choo se the cache nodes for downlink receiving, in add ition to the req uesting user . Based on the selection, the optimal power and transmission time c a n be d eriv ed. No te tha t this is an integrated continu ous and discrete o ptimization, its solution is summ arized below . Solution 1. Given the system state S f ,n , let d 1 , d 2 , .. be the indexes of ca che n o des, whose lar ge-scale a tten uation to the BS in the s -th segment is worse th an the ( f , n ) - th user . Moreo ver , without loss o f generality , it is assumed th a t ρ d 1 η d 1 f ,n,s ≤ ρ d 2 η d 2 f ,n,s ≤ ... ≤ ρ f ,n,s η f ,n,s . The optimal contr o l action for the s -th se gm e nt ( ∀ s ) c an be obtained below . • F or ea ch i , suppose the d i -th cache nod e ar e in volved for downlink r eceiving, the optimal power an d transmission time co ntr ol is give n by Q ∗ d i ,s ( S f ,n ) = min P f,n,s N f,n,s g f ,n,s ( S f ,n , Ω f ,n ) + X N ( λ f T f ,n ) N N ! e − λ f T f,n X j = { d 1 ,...,d i − 1 } e V N ( e S j,s f ) − e V N ( e S ∗ f ) s.t. R d i f ,n,s = R f / N S . The optimal solutio n , denoted as [ P d i f ,n,s , N d i f ,n,s ] , c a n be derived similar to Lemma 3. • Let d ∗ = arg min d i Q ∗ d i ,s , the solution of Pr oblem 5 is then given b y [ P ∗ f ,n,s , N ∗ f ,n,s ] = [ P d ∗ f ,n,s , N d ∗ f ,n,s ] . 6 Q u ( S f ,n ) = min Ω f ( S f,n ) X t g f ,n,t ( S f ,n , Ω f ,n ) + e V N R − n ( e S i,s f ) , s.t. R f ,n,t = R f / N S , ∀ t (9) Q i,s ( S f ,n ) = min Ω f ( S f,n ) X t g f ,n,t ( S f ,n , Ω f ) + e V N R − n ( e S ∗ f ) , s.t. R i f ,n,s = R f / N S and R f ,n,t = R f / N S , ∀ t (10) e V N R − n +1 ( e S i,s f ) = E η, ρ [ Q c | R f ,n,s ≤ R i f ,n,s ] P r ( R f ,n,s ≤ R i f ,n,s ) + E η, ρ [min { Q u , Q i,s }| R f ,n,s > R i f ,n,s ] P r ( R f ,n,s > R i f ,n,s ) (11) e V N R − n +1 ( e S f ,n ) ≤ e V N R − n +1 ( e S ∗ f ,n ) + X { ( i,s ) |∀ B i f,s ( e S f,n )=0 }  e V N R − n +1 ( e S i,s f ) − e V N R − n +1 ( e S ∗ f ,n )  (12) e V N R − n +1 ( e S f ,n ) ≥ e V N R − n +1 ( e S ∗ f ,n ) + X { ( i,s ) |∀ B i f,s ( e S f,n )=0 }  e V 1 ( e S i,s f ) − e V 1 ( e S ∗ f ,n )  (13) C. Boun d on V alue Fun ction Appr oximation In this section , we shall provide the bo und on g ap betwe e n the app r oximated value function and the actu a l value f unction. First of all, we in tr oduce the f ollowing bo unds on the actu al value f unction. Lemma 5 (Bounds o f V alue Fun ction) . The upper-bound in (12) holds fo r valu e fun ction e V N R − n +1 ( e S f ,n ) ( ∀ n ). Mor eover , if ther e is no overlap in th e service r egion of cache nodes, the lower -bou nd in ( 1 3) a lso holds for e V N R − n +1 ( e S f ,n ) . Pr oo f. Please ref e r to Ap pendix B. Notice that the prop o sed linear approxima tio n on value function is actua lly th e u pper-bound in (12), the gap b etween the appro ximated value function and the actua l value function e V N R − n +1 ( e S f ,n ) , d enoted as E N R − n +1 ( e S f ,n ) , is given by E N R − n +1 ( e S f ,n ) ≤ X { ( i,s ) |∀B i f,s ( e S f,n )=0 }  e V N R − n +1 ( e S i,s f ) − e V N R − n +1 ( e S ∗ f ) − e V 1 ( e S i,s f ) + e V 1 ( e S ∗ f )  . According to the de fin ition of value fu nction, the av erage system co st on th e f - th file with optimal con tr ol can be written as g ∗ f = X N R ( λ f T ) N R N R ! e − λ f T e V N R ( e S 0 f ) , (14) where e S 0 f denotes the system state with em pty buffer in all cache nodes. When the linear app r oximation in (8) is used, we ca n e valuate g ∗ f analytically with at most P N R ( λ f T ) N R N R ! e − λ f T E N R ( e S 0 f ) error . V I . S I M U L A T I O N In the simulation, the rad ius o f o ne ce ll is 50 0 m eters, ca c he nodes are rand omly deployed on the cell-edge region with a service rad ius o f 90 meters. Th e numb er of antennas at the BS is 8. The downlink path loss expo n ent is 3 . 5 . The file size is 14 0 Mb, and is fu rther divided in to 10 segments. The transmission band width is 20 MHz. The weigh ts on transmis- sion energy and time are w e = 1 and w t = 10 0 respectively . 1 2 3 4 5 6 7 8 Expectation of Request Number 20 40 60 80 100 120 140 160 180 200 220 Average Cost Baseline1 Baseline2 Proposed Fig. 1. The a verage total cost versus the expe ctati on of reque st times, where the number of cache nodes is 20. In ad dition to the pr oposed algo rithm, the perfo rmance of the following two baseline schemes is also compa r ed. Baseline 1. The BS on ly ensures the segment delive ry to the r eq uesting u sers in each transmission. Th e cache n odes with better chan nel con d ition to the BS can also deco de the file se g ments. Baseline 2. The BS ensur es that all the cache nod es can decode the downlink file in the fi rst transmission. Hence, all the cache no des can help to fo rwar d the file since th e seco nd file request. The perfo rmance o f the p roposed lo w-comp lexity algorith m is compar ed w ith th e a b ove two baselines in Fig.1, whe r e th e number of cache no des is 20 . It can be ob served that the propo sed algorithm is superior to the two baselines for any expected number of requests per file lifetime. M oreover , the Baseline 1 has be tter perfor mance than Baseline 2 wh en the popular ity of the file is h igh (larger expe c ted num b er o f file requests). The appr oximation err or on the value fun c tio n is illustrated 7 1 2 3 4 5 6 7 8 Total Transmission Times N R 0 50 100 150 200 250 300 Value Approximated Value Function Accurate Value Function Lower Bound Fig. 2. Illustrat ion of value function and its bounds. in Fig . 2, where the value fu nction ( av erage remain ing cost) for initial system state e S 0 f is plotted with different N R (total number of stages). Both upper and lower boun ds derived in Lemma 5 are plotted with the actual v alue f unction. It is shown that bo th bound s are tight, an d the approxim ation er r or is small. V I I . C O N C L U S I O N W e consider the downlink file tr ansmission with the assis- tance of cach e nodes in this p aper . Specifically , th e numb er of reque sts o f one file within its lifetim e is modele d as a Poisson point process, and the downlink resou rce minimization problem can be formulated as a Markov decision process with random num ber of stages. W e first prop ose a r evised Bellman’ s equation, wher e the o ptimal contro l p o licy can b e derived. In order to ad dress the curse of d imensionality , we also intro- duce a lo w-complexity sub-op tim al solution b ased on linear approx imation of value fun ction. T h e a pproxim ated value function can be calculated analytically , so that conv entional value iter ation can be eliminated. Finally , we derive a boun d on the gap betwe en the appr oximated value fun ction and the real value fu nction. It is shown by numerical simulation th a t the pro posed algor ithm (with the p roposed appr o ximated MDP approa c h ) ca n significantly reduce the resource co n sumption at th e BS. A P P E N D I X A : P R O O F O F L E M M A 3 First of all, we hav e the following approx imation o n the throug hput R f ,n,s . R f ,n,s ≈ N f ,n,s E h f,n,s  log 2  || h f ,n,s || 2 P f ,n,s N T σ 2 z  = N f ,n,s [ θ + log 2 ( P f ,n,s )] . W ith R f ,n,s = R f / N S , we have N f ,n,s = R f N S [ θ +log 2 ( P f,n,s )] . Hence th e o r iginal optimizatio n becomes min P f,n,s f ( P f ,n,s ) = min P f,n,s R f ( w e P f ,n,s + w t ) N S [ θ + log 2 ( P f ,n,s )] T aking first-ord e r deriv ativ e on f ( P f ,n,s ) , th e o ptimal tra n s- mission power P ∗ f ,n,s can be ob tained. A P P E N D I X B : P RO O F O F L E M M A 5 Compared with the co st e V N R − n +1 ( e S ∗ f ) , the a dditional c o st in e V N R − n +1 ( e S f ,n ) co mes fro m the fo llowing two cases: (1) users fall into th e coverage of certain ca c he no de, but this cache nod e d oes not have the desired file segments; (2) user s fall in to the r egion with out cache n odes, b ut the BS would like to spend more transmission resou rce so that some cache nod es can de c ode the downlink segmen ts together with the u ser s. Note that when the BS is tra nsmitting file segmen ts to on e cache nodes, some other cache nodes w ith better channel condition can also deco de the segments, w e ha ve P { ( i,s ) |∀ B i f,s ( e S f,n )=0 }  e V N R − n +1 ( e S i,s f ) − e V N R − n +1 ( e S ∗ f ,n )  is greater than e V N R − n +1 ( e S f ,n ) − e V N R − n +1 ( e S ∗ f ) . Moreover , the additional co st e V 1 ( e S i,s f ) − e V 1 ( e S ∗ f ,n ) comes from the extra transmission resource of the BS, which is exactly to serve one user in the coverage of the i -th cache node. Th us the lower -bou nd is also straig htforward. R E F E R E N C E S [1] X. W ang, M. Chen, T . T aleb, A. Ksentini, and V . C. M. Leung, “Cache in the air: exploiti ng content caching and deli very techn iques for 5g systems, ” IEE E Communications Magazine , vol. 52, no. 2, pp. 131–139, February 2014. [2] E. Batu, M. Bennis, and M. Debbah, “Cac he-enabl ed small cell net- works: Modeling and tradeof fs, ” in 2014 11th International Symposium on W ir eless Communication s Systems (ISWCS) , Aug 2014, pp. 649–653. [3] A. L iu and V . K. N. Lau, “Mixed-ti mescale precodi ng and cache control in cached mimo interfe rence netwo rk, ” IEEE T ransact ions on Signal Pr ocessing , vol. 61, no. 24, pp. 6320–6332, Dec 2013. [4] X. Li, X. W ang, and V . C. M. Leung, “W eighted netw ork traf fic of floading in cache-ena bled hete rogeneous networks, ” in 2016 IEEE Internati onal Confer ence on Communications (ICC) , May 2016, pp. 1– 6. [5] E. Bastug, M. Bennis, and M. Debbah, “Social and spatial proacti ve cachi ng for mobile data offloa ding, ” in 2014 IEEE International Con- fer ence on Communica tions W orkshops (ICC) , June 2014, pp. 581–586. [6] A. Check o, H. L. Christian sen, Y . Y an, L. Scolari , G. Kardara s, M. S. Berg er , and L. Dittmann, “Cloud ran for mobile netw orks a technol ogy ov ervie w , ” IEEE Communicatio ns Survey s T utorials , vol. 17, no. 1, pp. 405–426, Jan. 2015. [7] J. Koh, O. Simeone, R. T andon, and J. Kang, “Cloud-aided edge cachin g with wirele ss multicast fronthauli ng in fog radio ac cess networks, ” in 2017 IEEE W irele ss Communicat ions and Networki ng Confer ence (WCNC) , March 2017, pp. 1–6. [8] K. Poularaki s, G. Iosifidis, I. Pefkianakis, L. T assiulas, and M. May , “Mobile data offloadi ng through cachi ng in residentia l 802.11 wirele ss netw orks, ” IEEE T ransact ions on Network and Service Mana gement , vol. 13, no. 1, pp. 71–84, March 2016. [9] M. Ji, G. Caire, and A. F . Molisch, “Wir eless device -to-de vice caching netw orks: Basic principle s and system performance, ” IEEE Journal on Selec ted Areas in Communicat ions , vol. 34, no. 1, pp. 176–189, Jan 2016. [10] S. K. Jayawe era, Mark ov Decision P rocesse s . W i- ley T elec om, 2015, pp. 1–768. [Online]. A vai lable: http:/ /ieee xplore .ieee.org/xpl/articleDetails.jsp?arnumber=8045654 [11] U. T amm, “Some refelecti ons about the lambert w function as in verse of x*log(x), ” in 2014 Informatio n Theory and Applications W orkshop (IT A ) , Feb 2014, pp. 1–4.