Tertiary-Mode STAR-RIS for Secure NOMA: Integrating Transmission, Reflection, and Jamming

T ertiary-Mode ST AR-RIS for Secure NOMA: Inte grating T ransmission, Reflection, and Jamming Mansi Nema ∗ , Ku ntal Deka ∗ , Sanjee v Sharma † , and Tharmalingam Ratnarajah † ∗ Department of Electronics & Electrical Engineering, Indian Institute of T echnology Guwaha ti, India † Department of Electronics Engineering, Indian Institute of T echnology (BHU) V aranasi, India ‡ Dept. of Electrical and Computer Engg., San Diego State Uni versity , San Diego, USA emails: { n.mansi, kuntaldeka } @iitg.ac.in, sanjeev .ece@iitbhu.ac.in, t. r atnarajah@ieee.org Abstract —This paper inv estigates the physical lay er security of a non-orthogonal multiple access (NOMA) system a ssisted by a tertiary-mode simultaneously transmitting and re flecting reconfigurable intelligent surface (ST AR-RIS), which can per - fo rm transmission, r eflection, and jamming simultaneously . The system comprises a base station (BS) ser ving two users located on opposite sides of the ST AR-RIS, assuming perfect channel state informa tion (CSI) at the transmitter . T o enhance secrecy perfo r - mance, a subset of S T AR-RIS elements is adaptively configured fo r jamming. A penalty-based alternating optimization algorithm is de veloped to jointly optimize t h e BS’ s activ e beamf orming and the ST AR-RIS ’ s passive beamf orming and mode selection. Simulation results demo nstrate that the proposed design substan- tially impro ves the achieva ble sum rate and secrecy p erf ormance compared to conv entional RIS-assisted and no-RIS benchmarks, highlightin g th e potenti al of tertiary-mode ST AR-RIS for secure and efficient next-generation w i reless communications. Index T erms —Physical layer security , NOMA, RIS. I . I N T R O D U C T I O N R Econfigu rable intelligen t surfaces (RIS) have emerged as a promising techn o logy for next-genera tion wireless commun ication systems. They can significantly enhance both spectral and energy efficiency . An RIS consists o f m any low-cost passive elements that can adju st the amplitude and phase of incident electro m agnetic (EM) wa ves. By prop erly configur ing these elements, the RIS can r e c onfigur e the pro pa- gation environmen t to strengthen desired signals an d suppress interferen ce [1]. Unlike ac ti ve relays, RIS does not require power am plifiers or radio-freq uency (RF) chain s, which makes it simple, cost-effective, and energy- efficient. Owing to these advantages, RIS is rec ognized as an important enab ler for sustainable and energy -efficient sixth-ge n eration (6G) wireless networks. Although RIS offers m any b enefits, traditional implemen - tations operate only in the reflection m ode and can serve users located on a single side of the surface. This d irectional limitation r e stric ts its capab ility in providin g f ull-space cov- erage, p articularly in den se ur ban or non -line-of -sight (NLoS) en vironm ents. T o overcome th is limitation, the co ncept of a simultaneou sly transmitting and reflecting RIS (ST AR-RIS) has been in troduc ed [2]. A ST AR-RIS can both transmit and reflect incident signals, extend in g its coverag e to users o n bo th sides of th e surface. In recent years, ST AR-RIS has also attracted growing atten- tion in th e context o f physical layer security (PLS) and its inte- gration with non - orthog onal multiple access (NOMA) [3]–[ 5 ]. These stu d ies dem onstrate tha t ST AR-RIS can simu ltaneously enhance c overage, im prove spectral efficiency , and incr e a se the secrecy cap acity of wireless n etworks. Ho wev er , achieving optimal performan ce requires efficient configur ation of the ST AR-RIS, which remains a majo r challeng e. Th e associated optimization p roblem s are h ighly non-conve x and in volve the joint d esign of beamf orming , phase adjustme n t, and mode selection. T o address these ch allenges, se veral works hav e propo sed itera tive o ptimization frameworks such as alterna tin g optimization (A O) and p enalty-b ased method s [6] , [7], [8] , [9]. These ap proach e s achieve a reasonab le ba lance between system perform ance and computationa l co mplexity . Despite these a d vances, most existing ST AR-RIS studies employ energy splitting (ES) or time switching (TS) p rotoco ls, where th e transmitting and reflecting elemen ts are separated in the energy or time domain. Although ES and TS are simple to implem ent, they limit the flexibility an d adaptab ility of th e RIS elements, particu larly in dyna m ic wireless environments. This p aper adop ts a mo r e versatile mode switchin g (MS) protoco l, wh ere each ST AR-RIS element can in d epend e n tly operate in transm ission, reflection, or jamming m ode ac- cording to network and security requirements. W e consider a ST AR-RIS-assisted multiple-inp ut single-outp ut (MISO) NOMA system in which a multi-anten n a base station (BS) commun icates with two legitimate users lo cated o n op posite sides of the ST AR-RIS in the presence of an ea vesdropper . The ability of each ST AR-RIS element to dynam ic a lly select its operatin g m ode provides fine-graine d contr o l over the pro p- agation en v ironmen t, en hancing both thr ough put and secrecy . T o the best of the author s’ knowledge, this is the first work to analyze the secrecy and sum-rate per f orman c e of a ter tiary- mode ST AR-RIS-assisted NOMA system. The main co ntributions of this paper are: • A secure ST AR-RIS-assisted MISO-NOMA co mmunica- tion framework is propo sed, where each ST AR-RIS ele- ment can dynamically opera te in transmission, reflection, or jammin g mod e to jointly enhance the achiev able sum- rate and p hysical lay er security . • A joint optimization problem is formulated to m aximize the system sum-rate by jointly desig n ing the activ e beam- forming at the BS and the passive beamfo rming and mode selection at th e ST AR-RIS. • A penalty-based alternatin g optimization algorithm is developed to ef ficiently solve the r esulting no n-conve x problem , w h ich simultaneously hand le s continuo us an d discrete optimization variables in a un ified framew ork. I I . S Y S T E M M O D E L A N D P R O B L E M F O R M U L A T I O N Consider a tertiary-mo de MISO ST AR-RIS system f or se- cure NOMA comm unication s, as illustrated in Fig . 1. The BS, equip ped with N transmit anten nas, serves two legitimate users: one located on the transm ission side and the oth er o n the reflection side of th e ST AR-RIS. Each user is equ ipped with a single receive antenn a. I n ad dition, a passive eavesdropper is p o sitioned on the reflection side of the ST AR-RIS. It is assumed th at the eav esdropp er cannot intercep t the signal intended for the transmission-side user because ther e is no direct link b etween them. Fu rthermo re, the tr ansmission-side user does not h av e a line-o f-sight (L oS) chann el with the BS. Jam ing R f ection T ansmission Fig. 1: ST AR-RIS assis ted MISO NOMA com municatio n system. A. Channel Model Let H br ∈ C K × N denote th e ch a nnel matrix f r om the BS to the ST AR-RIS, wh ere K repr esents the numb er of ST AR- RIS elemen ts. Th e chan n els f rom the ST AR-RIS to User 1, User 2, and Eve are d e noted b y h r1 ∈ C K × 1 , h t2 ∈ C K × 1 , and h re ∈ C K × 1 , r espectively . The direct link s be tween the BS an d User 1 and b etween the BS and Eve ar e rep resented by h b1 ∈ C N × 1 and h b e ∈ C N × 1 , re sp ectiv ely . T h e direct BS–User 2 link is assum ed to be negligible due to se vere path loss or b lockage , which is typical in ST AR-RIS-assisted commun ication scenario s. User 1 is near er to the BS as shown in Fig. 1 . B. Signal Model The transmitted signal from th e BS is expressed as x = w 1 s 1 + w 2 s 2 , where s 1 and s 2 denote the info rmation - bearing symbols intended for User 1 and User 2, respectively , satisfying E [ | s 1 | 2 ] = E [ | s 2 | 2 ] = 1 . Here, w 1 and w 2 ∈ C N × 1 represent th e ac tive beamfor ming vecto rs for the two users. The total transmit po wer at the BS is constrained b y P max , such that k w 1 k 2 + k w 2 k 2 ≤ P max . Th e received signals at User 1, User 2, and Eve are : y 1 = ( h H r1 Θ r H br + h H b1 )( w 1 s 1 + w 2 s 2 ) + h H r1 z + n 1 , (1) y 2 = h H t2 Θ t H br ( w 1 s 1 + w 2 s 2 ) + n 2 , (2) y e = ( h H re Θ r H br + h H b e )( w 1 s 1 + w 2 s 2 ) + h H re z + n e , (3) where n 1 , n 2 , and n e ∼ C N (0 , σ 2 ) denote the ind epende n t additive wh ite Gaussian noise ( A WGN) at User 1, User 2, and Eve, respectively . The reflected jamming signal vector is denoted by z = Θ j H br x ∈ C K × 1 . C. ST AR-RI S Ope ration Modes Each ST AR-RIS elemen t can operate in one of th ree mu - tually exclusi ve mode s: reflection , tran smission, or jam ming. The correspond ing diagonal phase-shift ma trices are: Θ r = diag( r ) , r = [ β r, 1 e j θ r, 1 , . . . , β r,K e j θ r,K ] H , (4) Θ t = diag( t ) , t = [ β t, 1 e j θ t, 1 , . . . , β t,K e j θ t,K ] H , (5) Θ j = diag( j ) , j = [ β j, 1 e j θ j, 1 , . . . , β j,K e j θ j,K ] H , (6) where β r,k , β t,k , β j,k ∈ { 0 , 1 } denote the mode-selection co ef- ficients for th e k - th element, indicating whether it operates in reflection, tr a nsmission, or jamming m ode. T o e nsure exclusive operation , each elemen t satisfies β r,k + β t,k + β j,k = 1 . The correspo n ding ph ase shifts θ r,k , θ t,k , and θ j,k are contin uous variables within the rang e [0 , 2 π ) . W e use this simplified model to keep the o ptimization pr oblem tractable. Extending the analysis to a mo re general model with mu ltip le acti ve modes is an in teresting d irection for f u ture w ork. D. SINR a nd Achievable R ates Using the ab ove model, the various signal- to-interf e rence- plus-noise ratio (SI NR) term at each rece iver can be expressed as follows. For User 1 decoding its own signal: SINR 11 =   ( h H r1 Θ r H br + h H b1 ) w 1   2   ( h H r1 Θ r H br + h H b1 ) w 2   2 +   h H r1 Θ j H br w 1   2 + σ 2 . (7) For User 2 decoding its own signal: SINR 22 =   ( h H t2 Θ t H br ) w 2   2   ( h H t2 Θ t H br ) w 1   2 + σ 2 . (8) For User 1 decoding User 2’ s sign al (SIC process): SINR 12 =   ( h H r1 Θ r H br + h H b1 ) w 2   2   ( h H r1 Θ r H br + h H b1 ) w 1   2 +   h H r1 Θ j H br w 1   2 + σ 2 . (9) For Eve decoding U ser 1’ s signal: SINR e1 =   ( h H re Θ r H br + h H b e ) w 1   2 | h H re Θ j H br w 1 | 2 + | h H re Θ j H br w 2 | 2 + σ 2 . ( 10) The correspo nding achiev ab le rates (in bits per second per hertz) for the legitimate user a nd Eve are giv en by R 12 = log 2 (1 + SINR 12 ) , (11) R e1 = log 2 (1 + SINR e1 ) . (12) E. Pr oble m F ormula tion Our objective is to join tly optimize the BS beamfor ming vectors a n d the ST AR-RIS con figuration to m aximize the to tal achiev able sum -rate of both user s while maintaining secure transmission again st Eve. Let K denote the set of the RIS element indices i.e. K = { 1 , 2 , . . . , K } . The o ptimization problem can thus b e formulated as: (P1) : max w 1 , w 2 ,β z ,k R 11 + R 22 (13a) s.t. k w 1 k 2 + k w 2 k 2 ≤ P max , (13b) R 12 ≥ R 22 , (13c) β r,k + β t,k + β j,k = 1 , ∀ k ∈ K , (13d) β z ,k ∈ { 0 , 1 } , ∀ k ∈ K , z ∈ { r , t, j } , (13e) max R e1 ≤ τ , (13f) θ r,k , θ t,k , θ j,k ∈ [0 , 2 π ) , ∀ k ∈ K . (13g) The constraint (13b) en forces the BS tr a nsmit po wer limit, while (13c) ensur e s the feasibility of NOMA decoding order at User 1 for SIC. The con straints (13d )–(13e) define the ST AR- RIS element mode selection and exclusivity cond itio ns. The constraint (1 3f) limits the information leakage to Eve, i.e. the maximum allo wed value sho uld be less than o r e qual to τ . Finally , (13g) defines the feasible range for the ST AR-RIS phase-shift v ariables. I I I . P R O P O S E D S O L U T I O N The op tim ization problem in ( P1 ) is non-conve x due to variable coupling , namely the bilinear relation be twe en beam- forming vectors and ST AR-RIS phase ma tr ices, and d iscrete mode-selectio n coefficients β z ,k . T o address this, we adop t a dual stage penalty op timization that de c omposes the prob - lem in to two in terdepen dent subproblem s, ac ti ve and passiv e beamfor ming, so lved iterati vely by optimizing one variable set while keeping th e other fixed. A. Active Beamforming Optimization In the first stage of th e dual-stag e framew ork, we optimize the active beamfo rming vecto r s w 1 and w 2 at the BS while keeping the ST AR-RIS phase-shif t matrice s Θ r , Θ t , and Θ j fixed. T he objec tive is to maximize the achiev ab le su m-rate of legitimate users while en suring the ph ysical layer security constraint against the eavesdropper (Eve) is satisfied. 1) Simplification o f the o bj ective function by auxiliary variables r 11 , r 22 , and s : T o simplify the form ulation, we introdu c e the lifted matrices W 1 = w 1 w H 1 and W 2 = w 2 w H 2 , where W 1 , W 2 ∈ C N × N . These matrices represen t the covariance structure of the beamfo rming vectors, allowing the optimization to b e expre ssed in matrix form. Each W m is Herm itian, p ositiv e semidefinite, and rank- one. Th e activ e beamfor ming prob le m can be formulated as max W 1 , W 2 R 11 + R 22 (14a) s.t. W 1  0 , W 2  0 (14b) rank( W 1 ) = 1 (14c) rank( W 2 ) = 1 (14d) (13b) , (13c) , (13f) , (13g) . (14e) Note that the constra in ts in (14e) will b e considered in terms of the lifted matr ices later . The o b jective fu n ction R 11 + R 22 , r epresenting the to- tal achiev able sum -rate, is non-co n vex since each rate ter m R mm = log 2 (1 + SINR mm ) inv olves a no n-conve x SINR expression with a fractional q uadratic dependen ce on W 1 and W 2 . T o reformu late this in a con vex for m, we in- troduce slack v ariables r 11 and r 22 such that R mm = log 2 (1 + r mm ) and r mm ≤ SINR mm . Observe that the scaled o bjective function c a n b e written a s 1 2 ( R 11 + R 22 ) ≥ log  p (1 + R 11 )(1 + R 22 )  . Exploiting the mo noton icity of the lo garithmic f unction , the objecti ve can be eq u iv alently expressed in terms of another au xiliary variable s : s ≤ p (1 + r 11 )(1 + r 22 ) , (15 ) The o p timization pr o blem in terms of s w ill be shown later as (P2) in (3 1). Next, th e po wer constraint is expressed in term s of the lifted matrices as T r( W 1 ) + T r( W 2 ) ≤ P max . (16) The rank- one constraints in (14c) and (14 d) are relaxed resulting in semid efinite r elaxation ( SDR). T o simp lify c h annel depend encies, we define equ ivalent ef fective c hannel matrices as: H 1 = diag( h H r 1 ) H br h H b 1 , H 2 = diag( h H t 2 ) H br , and H e = diag( h H r e ) H br h H be . A n d lifted ma tr ices ar e: R = φ r φ H r , T = φ t φ H t and J = φ j φ H j where φ r , φ t , and φ j are the vectors co ntaining the diago nal entries of Θ r , Θ t , and Θ j , respectively . Using these definitions, the SIC dec o ding o rder constraint in ( 13c) can be expressed as T r( H 1 W 1 H H 1 R ) ≤ T r( H 2 W 2 H H 2 T ) . (17) 2) Simplification of the constraints (13c ) and (13f) by the slack variable r 12 : The constraint (13c) en sures the SIC de coding ord e r , i.e., User 1 must b e able to de c ode User 2’ s message before decoding its o wn. Mathematically , this translates to the inequality R 12 ≥ R 22 , or eq u iv a lently SINR 12 ≥ SINR 22 . Howe ver, the SINR expressions are no n - conv ex and cannot be dir ectly han d led by con vex solvers. T o conve xify this constraint, we in troduc e a slack v ariable r 12 ≤ SINR 12 , representing the effecti ve SINR of User 1 when decod in g User 2’ s signal. Using this variable, the rate- orderin g co nstraint in (13c ) can be equiv ale n tly expr essed as: r 12 ≥ r 22 , (18) where r 22 correspo n ds to User 2’ s ach iev able SINR. This transform ation preserves the SIC conditio n while en abling conv ex r eformu lation of the fractional SINR struc tu re. The instantaneous SINR for User 1 decoding User 2 ’ s message is gi ven by SINR 12 = T r  H 1 W 2 H H 1 R  T r  H 1 W 1 H H 1 R  + T r  H 1 W 1 H H 1 J  + σ 2 . (19) Re writing (19) u sing r 12 ≤ SINR 12 we g et r 12  T r  H 1 W 1 H H 1 R  + T r  H 1 W 1 H H 1 J  + σ 2  ≤ T r  H 1 W 2 H H 1 R  . (20) A , T r  H 1 W 2 H H 1 R  , (21) Γ , T r  H 1 W 1 H H 1 R  + T r  H 1 W 1 H H 1 J  + σ 2 . (22) Then (20) can be rewritten as r 12 Γ ≤ A. (23) Applying the arithmetic– geometric mean (A GM) in equality to the product ter m τ 12 Γ , for any auxiliary v ariable µ 12 = p r 12 τ , yields τ 12 Γ ≤ 1 2 ( ( µ 12 Γ) 2 +  r 12 µ 12  2 ) . (24) Combining (23) a nd (24), we ob tain 2 A ≥ ( µ 12 Γ) 2 +  r 12 µ 12  2 . (25) Substituting back A an d Γ , we get 2 T r  H 1 W 2 H H 1 R  ≥ h µ 12  T r  H 1 W 1 H H 1 R  + T r  H 1 W 1 H H 1 J  + σ 2 i 2 +  r 12 µ 12  2 (26) Now , fo r th e term  r 12 µ 12  2 , we add a variable a to ensure numerical stab ility . T o ensure co nvexity within the seco nd- order con e (SOC) framework, the fo llowing SOC c o nstraint is added:   [2 r 12 , µ 12 − a ] T   ≤ µ 12 + a. (27) Constraint (27) equiv alently e n forces the quadratic inequality 4 r 12 a ≤ ( µ 12 + a ) 2 − ( µ 12 − a ) 2 , main taining conve x feasibility . Throu g h th e com bination of the slack variable r 12 , the monoto nic transform a tion in (1 9), an d th e AGM–SOC refor- mulation in (26)–(27), th e o riginal n o n-convex constrain t in (13c) is successfully tr ansforme d into a con vex repr esentation. 2  T r( H 1 W 2 H H 1 R )  ≥ µ 2 12 Γ 2 + a, (28) The secrecy c onstraint in (13f) further e n sures limited informa tio n leakage to Eve, i.e., R e1 ≤ τ , w h ich translates to SINR e1 ≤ 2 τ − 1 . So, SINR e1 can be no w written as SINR e1 = T r  H e W 1 H H e R  T r( H e W 1 H H e J ) + σ 2 . (29) The fractiona l coupling is line arized in to the conv ex co n- straint T r( H e W 1 H H e R ) ≤ σ 2 + T r( H e W 1 H H e J ) , (30) ensuring that the jamm ing co mpon e nt domina te s at Eve. Finally , the con vexified active beamform ing sub prob lem is formu late d as (P2) : max W 1 , W 2 , r 12 , s s (31) s.t. (14b) , ( 15) , (1 6 ) , (17) , (18) , (2 7) , (28 ) , (30) . B. P assive Beamforming Optimization In th e seco nd stage, the ac tive beamf orming matrices W 1 and W 2 are fixed, and optimization is performed over the ST AR-RIS r e flection, transmission , and ja m ming matr ices R , T , and J , as well as th e mod e-selection coefficients β z ,k . Th e optimization problem is fo rmulated as max s, R , J , T ,β z ,r 12 s (32a) s.t. R  0 , J  0 , T  0 (32b) rank( R ) = 1 , rank( J ) = 1 , rank( T ) = 1 (32c) Diag( R ) = β r , Diag( J ) = β j , Diag( T ) = β t (32d) (13d) , (13e) , (15) , (17) , (27) , (28) , (30) , (32e) This problem remains non -conve x primarily due to two sources of no nlinearity : (i) the discrete bina r y natu re of the mode-selectio n variables β z ,k ∈ { 0 , 1 } , and (ii) th e rank-o ne constraints imposed o n the ST AR-RIS matrices R , T , J . 1) Binary mode-selection relaxa tion: The binary constraint β z ,k ∈ { 0 , 1 } is equiv alen tly represented by the quadratic inequality β 2 z ,k − β z ,k ≤ 0 with β z ,k ∈ [0 , 1 ] . T h e maximu m value of 0 is achieved wh en β z ,k = 0 or 1 . So , we want to maximize β 2 z ,k − β z ,k or min imize β z ,k − β 2 z ,k . T he no n-convex function β z ,k − β 2 z ,k is linearized using the first-ord er gradien t inequality [10] at the curre nt iterate β ( i ) z ,k : β z ,k − ( β z ,k ) 2 ≤ β z ,k − ( β ( i ) z ,k ) 2 − 2 β ( i ) z ,k ( β z ,k − β ( i ) z ,k ) , (33 ) which provides a tight conve x surrogate aro und the pr evious iteration. T o drive each β z ,k tow ard a b inary so lu tion, a penalty coefficient ζ > 0 is introduced into the o bjective fu nction. The coefficient is prog ressiv ely increased ( ζ ← ω ζ , ω > 1 ) after each iteration, ensuring that the solution conver ges to the discrete set { 0 , 1 } as the algo r ithm p roceeds. 2) Rank-one co nstraint relaxatio n: The constraints rank( R ) = rank( T ) = r ank( J ) = 1 are n on-co n vex an d cannot be hand led directly within a semidefinite prog ramming (SDP) framew ork. T o approximate the rank function, we employ the d ifference b e tween the nuclear nor m k Z k ∗ (sum o f singular values) an d the spectral norm k Z k 2 (largest sing ular value), i.e., k Z k ∗ − k Z k 2 ≥ 0 , Z ∈ { R , T , J } , (34) which is minimized if and on ly if Z is rank - one. The fun ction in (3 4) is concave with respect to Z because th e spectr al norm is con vex; therefo re, we linearize −k Z k 2 at the current iterate Z ( i ) using its subgrad ient. From conv ex a nalysis, the subgrad ie n t of the spectral norm at Z ( i ) is given by the outer product of the dominan t left and r ight singular vectors, denoted by λ 1 ( Z ( i ) ) λ 1 ( Z ( i ) ) H . Hence, applying the grad ient inequality , we get k Z k ∗ − k Z k 2 ≤ T r( Z ) + k Z ( i ) k 2 + T r  λ 1 ( Z ( i ) ) λ 1 ( Z ( i ) ) H ( Z − Z ( i ) )  (35) This first-order approx imation guarantee s c o n vexity at each iteration and en courag es low-rank (id eally rank-o ne) solu tions for the ST AR-RIS matrices. A pen alty coefficient ξ > 0 controls the streng th of this regularizatio n , en suring that the obtained solutions asympto tically satisfy the orig inal rank-one constraints. By co mbining the a bove co n vex surrog ates for th e b inary and rank -one constraints, the relaxed pro b lem (P 3) is trans- formed into a smoo th con vex pro gram as shown below: (P3) : max s, R , T , J , β z ,r 12 s − ζ X z ∈{ r,t,j } K X k =1 h ( β z ,k − β ( i ) z ,k ) 2 − ( β ( i ) z ,k ) 2 − 2 β ( i ) z ,k ( β z ,k − β ( i ) z ,k ) i − ξ X Z ∈{ R , T , J } h T r( Z ) + k Z ( i ) k 2 + Y ( Z , Z ( i ) ) i s.t. (13e) , (15) , (17) , (18) , (2 9 ) , (30 ) , (32) , (34 b) , (34c) (36) where, Y ( Z , Z ( i ) ) = T r  λ 1 ( Z ( i ) ) λ 1 ( Z ( i ) ) H ( Z − Z ( i ) )  As the p enalty co efficients ζ an d ξ increase, the solution conv erges toward the feasible set of the original mixed-integer non-co n vex problem. Problem s ( P2 ) and ( P 3 ) are solved alter- nately using the CVX toolbox . A f ter solving ( P 3 ), a one -hot encodin g step is applied wher e, for each elemen t k , the mode with the high est value amon g β r,k , β t,k , β j,k is set to 1 and the others to 0, en su ring tha t only on e mode is active at any time. The pen alty coefficients ζ and ξ are gradually inc reased by a scaling factor ω > 1 to en force convergence tow ard binary and rank-one solutio ns. This iterati ve A O proce ss continues until the improvement in th e objec ti ve value s falls below a predefined threshold ε . Algorithm 1 Dual Stage Penalty Op timization 1: Ini t ialize R , T , J , β r , β t , β j , ξ, ζ , µ 12 a , µ 12 b , ε, set i = 1 2: Repeat 4: Set j = 1 3: Repeat 5: Solv e activ e beamforming ( P2 ) in (31) 6: Update µ ( i +1) 12 a = p r 12 τ and increment j 7: Until | s ( i +1) − s ( i ) | ≤ ε, 8: Output s ∗ , W ∗ 1 , W ∗ 2 with optimized v alues after stage 1 9: Calculate R 1 = log 2  ( s ∗ ) 2  10: Set k = 1 11: Repeat 12: Solv e passive beamforming ( P3 ) i n (36) 13: Upd ate µ ( i +1) 12 b and increment k 14: Upd ate ξ = ω ξ 15: Upd ate ζ = ωζ 16: Unti l | s ( i +1) − s ( i ) | ≤ ε, 17: Outpu t s ∗ , β ∗ r , β ∗ t , β ∗ j , R ∗ , T ∗ , J ∗ , with optimized v alues after stage 2. 18: Upd ate β r , β t , β j by using one hot encoding 19: Calculate R 2 = log 2  ( s ∗ ) 2  20: Until | R 2 − R 1 | ≤ ε I V . R E S U LT S A N D D I S C U S S I O N T ABLE I pr esents the simulation parameters considered in this p aper . Unless stated oth e r wise, th e default system setup assumes a ST AR-RIS with K = 30 elements, a BS equippe d with N = 2 antennas, an d per f ect CSI at the transmitter is assumed. Fig. 2 co mpares the ac h iev able sum- rate of the p roposed tertiary - mode ST AR-RIS with jam ming against several benchmark schemes, including conventional RIS, ST AR-RIS without jamming , traditiona l RIS with jam- ming, an d a baseline w ith out RIS. It is obser ved that the propo sed system consistently outp erform s all baselines across the entire tr ansmit power range (0–4 0 dBm). The ST AR-RIS with in tegrated jamm ing achieves a notable imp rovement of approx imately 1 bit/s/Hz in secrecy rate over conventional RIS-assisted systems. Th is perfo rmance gain ar ises from the intelligent mode switching capability of th e ST AR-RIS, which dynamica lly allocates eleme n ts among transmission, reflec- tion, and jammin g fun ctions to maximize secr ecy per f orman c e. The tradition al RIS with jamming and ST AR-RIS yield similar results. Th eir p erform ance is better than that of the trad itional RIS. Finally , systems without RIS exhib it the lowest rates due to the ir inab ility to contro l chann el prop a gation, confirming th e significant benefit of inco rporatin g r e configur able surfaces. T ABLE I: Simulation Parameters Param. Descripti on V alue α LOS Path loss exponents (Rayleig h) 2 α NLOS r Path loss exponents (Rician) 2.8 α NLOS t Path loss exponents (Rician) 3 K rician Ricia n fac tor 1 λ 0 Path loss at 1 meter − 30 dB ε Con vergen ce threshold 0 .01 ζ , ξ Penalty fac tors 0.0001 ω Scaling fac tor 1.5 τ Info leaka ge tolerance 1 bit /s/Hz σ 0 Noise po wer − 90 dBm 0 10 20 30 40 0 2 4 6 8 10 12 14 T ransmit Power (dBm) Rate (bit/s/Hz) ST AR-RIS with Jamming ST AR-RIS Traditi onal RIS Traditi onal RIS with Jamming Wit hout RIS Fig. 2: Sum-rate versus transmit power for different sche m es ( K = 30 ). Fig. 3 analy zes the effect o f v arying the numbe r of ST AR- RIS elements ( K = 20 , 30 , 4 0 , 50 , 6 0 ) o n the achievable sum- rate. As expected, increasing K improves the per forman ce since a larger number o f elements pr ovides finer control over the in cident and reflected wavefronts, leadin g to hig her beamfor ming gain and better interfere nce man agement. Th e improvement becom e s less significant beyo n d K = 50 , sug- gesting a dimin ish in g retur n in r ate en hancemen t as the surface size grows. This observation highlig hts a key trade-off between perfor mance ga in and hardware comp lexity in p r actical ST AR- RIS deployments. 0 10 20 30 40 0 2 4 6 8 10 12 14 16 18 20 T ransmit P o wer (dBm) Rate (bit/s/Hz) K = 20 K = 30 K = 40 K = 50 K = 60 Fig. 3 : Impact o f ST AR-RIS elements K o n the sum-rate versus tra n smit po wer performance . Fig. 4 shows the adap ti ve mode allocation of ST AR-RIS el- ements (reflection, transmission, an d jamming) u nder dif ferent transmit power levels. At lo wer transmit powers, a m ajority of the elements ope r ate in the r e flection and transmission mod e s to strengthen the useful signa ls fo r legitimate users. Howe ver , as the transmit power incr eases, the system allocates a larger fraction of elements to the jamming mode to ef fectively sup - press potential eavesdropping . The p ropo sed mode-switchin g framework enab les the ST AR-RIS to autonomo usly balan ce between data forwarding and interferen c e generation. 0 10 2 0 30 40 0 10 20 30 15 20 24 23 21 12 7 6 6 7 3 3 0 1 2 T ransmit po wer (dBm) Number of elements Reflection T ransmission Jamming Fig. 4 : Distribution of ST AR-RIS mode s ( reflection, transm is- sion, jamming) versu s tran smit p ower . Fig. 5 demo nstrates the co n vergence per forman ce of the p r o- posed penalty- based altern ating optimization algorithm. The figure com p ares th e conv ergence curves f or active beamf orm- ing, passive b e amform ing, and the overall joint optimiza tio n process at a transmit power of 40 dBm. It is evident th at the algorithm converges within a f ew itera tio ns (typically less than 10) for all sub problem s, indicating compu ta tio nal efficiency . The overall ra te improvement stabilizes after ap proxim ately six iterations, confirm ing the stability and robustness of th e propo sed optimization framework. T his demonstrates that th e penalty-b ased m ethod provides fast and stable con vergen ce while main taining a near-optimal trade- off between ra te m ax- imization and secrecy constraints. 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Number of Iterations Rate (bit/s/Hz) Conv ergence for Active Beamforming Conv ergence for Passive Beamforming Conv ergence for overall algorithm Fig. 5: Con vergence r ate of the pro posed alg orithm for the ST AR-RIS system at 4 0 dBm. V . C O N C L U S I O N This paper presented a secu r e ST AR-RIS-aid ed NOMA system where each ST AR-RIS elem ent operates in reflection, transmission, or jammin g m o de. By jointly optimizing activ e beamfor ming at the BS and passive beam formin g with mod e selection at th e ST AR-RIS, significant improvements in se- crecy a nd spectral ef ficiency were a chieved u nder perfect CSI. Simulations showed that com bining reflection and jamming provides sup erior perform ance comp ared to con ventional IRS- based an d non- optimized systems. Future work c an extend th is framework to scenarios with imperfec t CSI, user mobility , or distributed ST AR-RIS, and inves tigate learning -based meth ods for real-time ad a ptation in d ynamic en v ironmen ts. R E F E R E N C E S [1] Q. W u, S. Z hang, B. Zheng, C. Y ou, a nd R. Zhan g, “Inte lligent reflecting surfac e-aide d wir eless communications: A tutorial , ” IEEE T rans. on Commun. , vo l. 69, no. 5, pp. 3313–3351, 2021. [2] X. Mu, J. Xu, Z. W ang, and N. Al-Dhahir , “Simultaneously transmitti ng and reflect ing surf aces for ubiquitous next generation mult iple access in 6G and be yond, ” Proc eedings of the IEE E , 2024. [3] H. Jia, L. Ma, and S. V alaee, “ST AR-RIS enable d do wnlink secure NOMA netw ork under imperfe ct CSI of eav esdroppe rs, ” IEEE Commun. Lett. , vol . 27, no. 3, pp. 802–806, 2023. [4] Y . Han, N. Li, Y . Liu, T . Zhang, and X. T ao, “Arti ficial noise aided secure NOMA communications in ST AR-RIS networks, ” IEEE W ir eless Commun. Lett. , vol. 11, no. 6, pp. 1191– 1195, 2022. [5] Z. Zhang, J. Chen, Y . Liu, Q. Wu , B. He, and L. Y ang, “On the secrecy design of ST AR-RIS assisted uplink NOMA networks, ” IEEE T rans. on W irel ess Commun. , vol. 21, no. 12, pp. 11 207–11 221, 2022. [6] M. Ji, J. Chen, L. Lv , Q. Wu, Z. Ding, and N. Al-Dhahi r , “Secure NOMA systems with a dual-functi onal RIS: Simultaneou s information relayi ng and jamming, ” IEE E T rans. on Commun . , 2023. [7] H. Han, Y . Cao, N. D eng, C. Xing, N. Zhao, Y . Li, and X. W ang, “Secure transmission for ST AR-RIS aided NOMA against inte rnal eav esdroppi ng, ” IEEE T rans. on V eh. T ec hnol. , vol. 72, no. 11, pp. 15 068–15 073, 2023. [8] S. Abeywic krama, R. Z hang, Q. W u, and C. Y uen, “Intelli gent reflecting surfac e: Practica l phase shift model and beamforming optimizat ion, ” IEEE T rans. on Commun. , vol. 68, no. 9, pp. 5849–5863, 2020 . [9] H. Niu, Z. Chu, F . Zhou, and Z. Z hu, “Simultaneous transmission and reflection reconfigurable intell igent surf ace assisted secrec y MISO netw orks, ” IEEE Commun. Lett. , vol. 25, no. 11, pp. 3498–3502, 2021. [10] D. P . Bertsekas, Nonlinear P r ogr amming , 3rd ed. Belmont, Mas- sachusett s: Athena Scientific, 2018.