Evolutionary and deep reinforcement learning algorithms for optimizing the lifetime of wireless sensor networks

YROXWLRQDU\ DQG 'HHS 5HLQIRUFHPHQW /HDUQLQJ $OJRULWKPV IRU 2SWLPL]LQJ WKH /LIHWLPH RI :LUHOHVV 6HQVRU 1HWZRUNV$EVWUDFW ,Q UHFHQW \HDUV ZLUHOHVV VHQVRU QHWZRUNV :61V KDYH EHFRPH DQ HVVHQW

)"/0* 6/*7&34*5: 0' 4$*&/$& "/% 5&$)/0-0(:

."45&3 5)&4*4

&WPMVUJPOBSZ BOE %FFQ 3FJOGPSDFNFOU -FBSOJOH "MHPSJUINT GPS 0QUJNJ[JOH UIF -JGFUJNF PG 8JSFMFTT 4FOTPS /FUXPSLT

#6* )0/( /(0$

QJRFEKP#VLVKXVWHGX

)"/0* 6/*7&34*5: 0' 4$*&/$& "/% 5&$)/0-0(:

."45&3 5)&4*4

&WPMVUJPOBSZ BOE %FFQ 3FJOGPSDFNFOU -FBSOJOH "MHPSJUINT GPS 0QUJNJ[JOH UIF -JGFUJNF PG 8JSFMFTT 4FOTPS /FUXPSLT

#6* )0/( /(0$

QJRFEKP#VLVKXVWHGX

4JHOBUVSF PG BEWJTPS

ĈӅWjLOXұQYăQCác giҧi thuұt tiӃn hoá và hӑFWăQJFѭӡQJÿӇ tӕLѭXWKӡi gian

sӕng cӫa mҥng cҧm biӃn không dây

Chuyên ngành: KRDKӑFGӳOLӋX(Elitech)

0mVӕ69: 20212155M

7iFJLҧ1JѭӡLKѭӟQJGүQNKRDKӑFYj+ӝLÿӗQJFKҩPOXұQYăQ[iFQKұQWiFJLҧ ÿmVӱDFKӳDEәVXQJOXұQYăQWKHRELrQEҧQKӑS+ӝLÿӗQJQJj\ 28/04/2023 YӟLFiF QӝLGXQJVDX

- +LӋXFKӍQKWrQFKѭѫQJ YjFKѭѫQJ ÿӇOjPU}KѫQQӝLGXQJWUuQKEj\

- 7KrPFiFWKҧROXұQӣFKѭѫQJQKҵPOjPU}KѫQFiFVӵNKiFELӋWWURQJÿyQJ JySӣFKѭѫQJVRYӟLÿӗiQWӕWQJKLӋSNӻVѭFӫDFQJWiFJLҧ

- 7KrPFiFWKҧROXұQӣFKѭѫQJYjFKѭѫQJÿӇOjPU}ÿӝQJOӵFYjVӵNKiFELӋW FӫDWKXұWWRiQÿӅ[XҩWVRYӟLFiFWKXұWWRiQWUѭӟFÿy

- 7KrPFiFWKҧROXұQYӅKҥQFKӃFӫDFiFJLҧÿӏQKVRYӟLP{LWUѭӡQJWKӵFWӃWURQJ FiFEjLWRiQӣFKѭѫQJYjFKѭѫQJ

- 5jVRiWKLӋXFKӍQKFiFOӛLVRҥQWKҧR

1Jj\WKiQJQăP2023

&+Ӫ7ӎ&++Ӝ,ĈӖ1*

 'HFODUDWLRQV'LVFORVXUHV 

, ± %XL +RQJ 1JRF ± KHUHE\ ZDUUDQWV WKDW WKH ZRUN DQG SUHVHQWDWLRQ LQ WKLV WKHVLVDUH SHUIRUPHG E\ P\VHOI XQGHU WKH VXSHUYLVLRQ RI 'U 1JX\HQ 3KL /H $OO UHVXOWVSUHVHQWHG LQ WKLV WKHVLV DUH WUXWKIXO DQG DUH QRW FRSLHG IURP DQ\ RWKHU ZRUNV $OOUHIHUHQFHV LQ WKLV WKHVLV  LQFOXGLQJ LPDJHV WDEOHV ILJXUHV DQG TXRWHV  DUH FOHDUO\DQG IXOO\ GRFXPHQWHG LQ WKH ELEOLRJUDSK\ , ZLOO WDNH IXOO UHVSRQVLELOLW\ IRU HYHQRQH FRS\ WKDW YLRODWHV VFKRRO UHJXODWLRQV

, ZRXOG OLNH WR H[SUHVV P\ JUDWLWXGH WR DOO WKRVH ZKR KDYH PRWLYDWHG DQG DLGHG PH LQDFKLHYLQJ WKLV VLJQLILFDQW PLOHVWRQH LQ P\ OLIH 0\ KHDUWIHOW DSSUHFLDWLRQ JRHV RXW WR P\WKHVLV DGYLVRUV 'U 1JX\HQ 3KL /H DQG 3URI 'R 3KDQ 7KXDQ ZKRVH HQFRXUDJHPHQW DQGJXLGDQFH ZHUH LQYDOXDEOH LQ VKDSLQJ WKH UHVHDUFK TXHVWLRQV DQG PHWKRGRORJ\ :LWKRXWWKHLU VXSHUYLVLRQ DQG LQVLJKWIXO VXJJHVWLRQV FRPSOHWLQJ WKLV WKHVLV ZRXOG KDYH EHHQ DQLQVXUPRXQWDEOH FKDOOHQJH

, RZH D GHEW RI JUDWLWXGH WR P\ DGYLVRU DW 9LQ$, 'U 1JX\HQ 9LHW $QK IRU KLV XQZDYHULQJ VXSSRUW WKURXJKRXW P\ WZR \HDUV DW 9LQ$, +LV H[SHUWLVH DQG JXLGDQFH KDYHHQDEOHG PH WR QRW RQO\ FRPSOHWH WKLV WKHVLV EXW DOVR JDLQ WKH NQRZOHGJH DQG VNLOOV QHFHVVDU\ IRU P\ IXWXUH FDUHHU %H\RQG DFDGHPLF UHVHDUFK KH ZDV DOZD\V DYDLODEOH WR SURYLGHDGYLFH DQG JXLGDQFH RQ YDULRXV DVSHFWV RI P\ SHUVRQDO DQG SURIHVVLRQDO GHYHORSPHQW +LVVXSSRUW KDV EHHQ D VRXUFH RI LQVSLUDWLRQ DQG PRWLYDWLRQ IRU PH WKURXJKRXW P\ MRXUQH\

$GGLWLRQDOO\ 7KH LQYDOXDEOH IHHGEDFN DQG VXJJHVWLRQV SURYLGHG E\ 3URI +X\QK 7KL7KDQK %LQK DQG 'U 1JX\HQ 7KL 7DP ZHUH HVVHQWLDO WR WKH FRPSOHWLRQ RI WKLV WKHVLV 7KHLUH[WHQVLYH H[SHUWLVH LQ WKH ILHOG ZDV FUXFLDO WR WKH GHYHORSPHQW RI YDULRXV DVSHFWV RI WKLVZRUN DQG , DP GHHSO\ JUDWHIXO IRU WKHLU VXSSRUW DQG JXLGDQFH WKURXJKRXW WKH SURFHVV:LWKRXW WKHLU FRQWULEXWLRQV WKLV WKHVLV ZRXOG QRW KDYH EHHQ FRPSOHWHG

, DOVR ZLVK WR DFNQRZOHGJH P\ IULHQGV DW +867 DQG 9LQ$, ZKRVH XQZDYHULQJ VXSSRUWDQG FRPSDQLRQVKLS KDYH EHHQ D VRXUFH RI VWUHQJWK DQG PRWLYDWLRQ WKURXJKRXW P\ WLPH

DW WKH XQLYHUVLW\ DQG FRPSDQ\ 7KH PHPRULHV RI SOD\LQJ JDPHV ZDWFKLQJ PRYLHV DQGHQMR\LQJ IRRWEDOO PDWFKHV DUH SULFHOHVV DQG KDYH NHSW PH JRLQJ

/DVWO\ , GHGLFDWH WKLV WKHVLV WR P\ IDPLO\ P\ SDUHQWV ZKR KDYH DOZD\V HQFRXUDJHG

PH WR SXUVXH P\ GUHDPV P\ JLUOIULHQG ZKR KDV VWRRG E\ PH GXULQJ P\ PRVW GLIILFXOWPRPHQWV OLVWHQLQJ DQG HPSDWKL]LQJ ZLWK P\ JULHYDQFHV DERXW WKH ZRUOG DQG P\ VLVWHUZKR PD\ VRPHWLPHV EH D QXLVDQFH EXW DOZD\V DGG FRORU WR P\ OLIH 7RJHWKHU WKH\ KDYHVKDSHG WKH SHUVRQ , DP WRGD\

L

(YROXWLRQDU\ DQG 'HHS 5HLQIRUFHPHQW /HDUQLQJ $OJRULWKPV IRU 2SWLPL]LQJ WKH /LIHWLPH RI :LUHOHVV 6HQVRU 1HWZRUNV

$EVWUDFW

,Q UHFHQW \HDUV ZLUHOHVV VHQVRU QHWZRUNV :61V KDYH EHFRPH DQ HVVHQWLDO SDUW RIPDQ\ FLYLOLDQ DSSOLFDWLRQV VXFK DV VPDUW DJULFXOWXUH KHDOWK PRQLWRULQJ DQG VPDUW FLWLHV+RZHYHU WKH OLPLWHG HQHUJ\ FDSDFLW\ RI WKH VHQVRUV SRVHV D VLJQLILFDQW FKDOOHQJH WR HQVXULQJ FRQWLQXRXV VXUYHLOODQFH ,Q WKLV WKHVLV ZH VWXG\ WZR HPHUJLQJ WHFKQLTXHV WR SURORQJ WKH QHWZRUN¶V OLIHWLPH HQHUJ\HIILFLHQW URXWLQJ YLD UHOD\ QRGH SODFHPHQW DQG URXWLQJVWUDWHJLHV IRU WKH PRELOH FKDUJHU LQ ZLUHOHVV UHFKDUJHDEOH VHQVRU QHWZRUNV :561V 7KH ILUVW SUREOHP LQYROYHV RSWLPL]LQJ WKH URXWLQJ SURWRFRO E\ GHSOR\LQJ QRQVHQVLQJUHOD\ QRGHV 51V  5HFHQW UHVHDUFK HLWKHU IRFXVHG RQ ILQGLQJ WKH PLQLPXP QXPEHU RI51V UHTXLUHG WR UHGXFH GHSOR\PHQW FRVWV RU GHYHORSLQJ HIILFLHQW URXWLQJ VFKHPHV WR UHGXFH HQHUJ\ XVH +RZHYHU VWULNLQJ D EDODQFH EHWZHHQ WKHVH FULWHULD UHPDLQV D VLJQLILFDQWFKDOOHQJH 7R DGGUHVV WKLV LVVXH ZH SURSRVH D PXOWLREMHFWLYH DSSURDFK WR FRQVWUXFWLQJ

DQ HIILFLHQW FRPPXQLFDWLRQ VWUXFWXUH ZLWK WKH OHDVW SRVVLEOH QXPEHU RI 51V WKHUHE\ SURORQJLQJ WKH QHWZRUN¶V OLIHWLPH ZKLOH HQVXULQJ LWV FRQQHFWLYLW\ DQG UHOLDELOLW\ :H FRQGXFWH[WHQVLYH H[SHULPHQWV WR GHPRQVWUDWH WKH HIIHFWLYHQHVV RI RXU PHWKRG DQG VKRZ WKDW LWSURYLGHV D EHWWHU WUDGHRII FRPSDUHG WR H[LVWLQJ DOJRULWKPV

,Q WKH VHFRQG SUREOHP ZH IRFXV RQ DGGUHVVLQJ WKH FKDOOHQJHV IDFHG LQ :561V ZKHUH

D PRELOH FKDUJHU FDQ EH HPSOR\HG WR PRYH DURXQG DQG FKDUJH WKH VHQVRUV ([LVWLQJ DSSURDFKHV VWUXJJOH WR GHVLJQ DQ RSWLPDO FKDUJLQJ SDWK IRU WKH PRELOH FKDUJHU ZKLOH DFFRXQWLQJ IRU WKH XQFHUWDLQWLHV DULVLQJ LQ WKH QHWZRUN ZKLFK FRXOG FRPH IURP QRGH IDLOXUHV RUGHSOR\PHQWV 7R RYHUFRPH WKLV FKDOOHQJH ZH SURSRVH D QRYHO FKDUJLQJ VFKHPH WKDW XVHV DGHHS UHLQIRUFHPHQW OHDUQLQJ '5/ DSSURDFK WR JXLGH WKH PRELOH FKDUJHU DGDSWLYHO\ 2XUDSSURDFK HQDEOHV WKH PRELOH FKDUJHU WR DGDSW WR VSRQWDQHRXV FKDQJHV LQ WKH QHWZRUN WRSRORJ\ 7KH H[SHULPHQWV VKRZ WKH VXSHULRULW\ RI RXU PRGHO FRPSDUHG WR H[LVWLQJ RQGHPDQGPHWKRGV LQ SURORQJLQJ WKH QHWZRUN OLIHWLPH

2XU SURSRVHG VROXWLRQV IRU WKH WZR SUREOHPV RI UHOD\ QRGH SODFHPHQW DQG PRELOHFKDUJHU URXWLQJ FDQ VLJQLILFDQWO\ SURORQJ WKH QHWZRUN¶V OLIHVSDQ DQG UHGXFH PDLQWHQDQFHFRVWV 7KH SRWHQWLDO IRU FRPELQLQJ WKHVH WZR WHFKQLTXHV LQ :561V WR IXUWKHU HQKDQFH WKHQHWZRUN¶V VXVWDLQDELOLW\ DQG UHOLDELOLW\ LV DQ LQWHUHVWLQJ IXWXUH UHVHDUFK GLUHFWLRQ $GRSWLQJWKHVH VROXWLRQV HVSHFLDOO\ QHZ DGYDQFHPHQWV LQ GHHS UHLQIRUFHPHQW OHDUQLQJ FDQ IDFLOLWDWHWKH GHYHORSPHQW RI PRUH HIILFLHQW DQG HIIHFWLYH :61V HQDEOLQJ XV WR EHWWHU PRQLWRU DQGPDQDJH YDULRXV V\VWHPV DQG SURFHVVHV LQ RXU GDLO\ OLYHV

LL

&iF JL̫L WKX̵W WL͇Q KRi Yj K͕F WăQJ F˱ͥQJ ÿ͋ W͙L ˱X WKͥL JLDQ

V͙QJ FͯD P̩QJ F̫P EL͇Q NK{QJ Gk\

7yP WҳW OXұQ YăQ

7URQJ QKӳQJ QăP JҫQ ÿk\ PҥQJ FҧP ELӃQ NK{QJ Gk\ ÿm WUӣ WKjQK PӝW SKҫQ FҫQ WKLӃWWURQJ QKLӅX ӭQJ GөQJ GkQ Vӵ QKѭ JLiP ViW VӭF NKӓH Q{QJ QJKLӋS Yj WKjQK SKӕ WK{QJPLQK 7X\ QKLrQ OѭӧQJ QăQJ OѭӧQJ KҥQ FKӃ ÿѭӧF FjL ÿһW WURQJ FiF FҧP ELӃQ ÿһW UD WKiFKWKӭF OӟQ WURQJ YLӋF ÿҧP EҧR GӏFK Yө WURQJ FiF ӭQJ GөQJ \rX FҫQ JLiP ViW OLrQ WөF Yj YӟLWҫQ VXҩW FDR 7URQJ OXұQ YăQ Qj\ FK~QJ W{L QJKLrQ FӭX KDL SKѭѫQJ SKiS PӟL ÿӇ NpR GjLWXәL WKӑ PҥQJ ÿӏQK WX\ӃQ WLӃW NLӋP QăQJ OѭӧQJ WK{QJ TXD ÿһW FiF Q~W WUXQJ JLDQ Yj FKLӃQOѭӧF ÿӏQK WX\ӃQ FKR Eӝ VҥF GL ÿӝQJ WURQJ PҥQJ FҧP ELӃQ Fy WKӇ VҥF OҥL NK{QJ Gk\

%jL WRiQ ÿҫX WLrQ OLrQ TXDQ ÿӃQ WӕL ѭX KyD JLDR WKӭF ÿӏQK WX\ӃQ EҵQJ FiFK WULӇQ NKDLFiF Q~W WUXQJ JLDQ NK{QJ FҧP ELӃQ &iF QJKLrQ FӭX JҫQ ÿk\ WұS WUXQJ YjR WuP Vӕ OѭӧQJWӕL WKLӇX Q~W WUXQJ JLDQ FҫQ WKLӃW ÿӇ JLҧP FKL SKt WULӇQ NKDL KRһF SKiW WULӇQ FiF FKLӃQ OѭӧFÿӏQK WX\ӃQ ÿӇ JLҧP QăQJ OѭӧQJ Vӱ GөQJ 7X\ QKLrQ WuP Vӵ FkQ EҵQJ JLӳD FiF WLrX FKt Qj\YүQ Oj PӝW WKiFK WKӭF OӟQ ĈӇ JLҧL TX\ӃW YҩQ ÿӅ Qj\ FK~QJ W{L ÿӅ [XҩW PӝW SKѭѫQJ SKiS

ÿD PөF WLrX ÿӇ [k\ GӵQJ PӝW FҩX WU~F WUX\ӅQ WK{QJ KLӋX TXҧ YӟL Vӕ OѭӧQJ Q~W WUXQJ JLDQ tWQKҩW Fy WKӇ Wӯ ÿy NpR GjL WXәL WKӑ PҥQJ WURQJ NKL ÿҧP EҧR NӃW QӕL FӫD Qy &K~QJ W{L WLӃQKjQK FiF WKӱ QJKLӋP ÿӇ FKӭQJ PLQK WtQK KLӋX TXҧ FӫD SKѭѫQJ SKiS FӫD FK~QJ W{L Yj FKRWKҩ\ Qy FXQJ FҩS Vӵ FkQ EҵQJ WӕW KѫQ VR YӟL FiF WKXұW WRiQ KLӋQ Fy

Ӣ EjL WRiQ WKӭ KDL FK~QJ W{L WұS WUXQJ YjR YLӋF JLҧL TX\ӃW FiF WKiFK WKӭF WURQJ PҥQJFҧP ELӃQ NK{QJ Gk\ VҥF OҥL QѫL PӝW Eӝ VҥF GL ÿӝQJ Fy WKӇ ÿѭӧF Vӱ GөQJ ÿӇ GL FKX\ӇQ YjVҥF FiF FҧP ELӃQ %jL WRiQ TXDQ WUӑQJ Oj WKLӃW NӃ PӝW FKLӃQ OѭӧF VҥF KLӋX TXҧ FKR Eӝ VҥF

GL ÿӝQJ WURQJ NKL SKҧL ÿӕL PһW YӟL FiF \ӃX Wӕ EҩW ÿӏQK [ҧ\ UD WURQJ PҥQJ QKѭ Q~W Eӏ OӛLKRһF Q~W PӟL ÿѭӧF WULӇQ NKDL &K~QJ W{L ÿӅ [XҩW PӝW FKLӃQ OѭӧF VҥF PӟL Vӱ GөQJ SKѭѫQJSKiS KӑF WăQJ FѭӡQJ VkX ÿӇ KѭӟQJ GүQ Eӝ VҥF GL ÿӝQJ PӝW FiFK OLQK KRҥW 3KѭѫQJ SKiSFӫD FK~QJ W{L FKR SKpS Eӝ VҥF GL ÿӝQJ WKtFK QJKL YӟL FiF WKD\ ÿәL EҩW QJӡ WURQJ FҩX WU~FPҥQJ QKӡ FiF Fѫ FKӃ PӟL WURQJ KӑF Pi\ &iF WKӱ QJKLӋP FKR WKҩ\ WtQK ѭX YLӋW FӫD P{KuQK FӫD FK~QJ W{L VR YӟL FiF SKѭѫQJ SKiS KLӋQ WҥL WURQJ YLӋF NpR GjL WXәL WKӑ PҥQJ

&iF JLҧL SKiS ÿӅ [XҩW FӫD FK~QJ W{L FKR KDL YҩQ ÿӅ YӅ Yӏ WUt Q~W WUXQJ JLDQ Yj ÿӏQKWX\ӃQ Eӝ VҥF GL ÿӝQJ Fy WKӇ NpR GjL WXәL WKӑ FӫD PҥQJ Yj JLҧP FKL SKt EҧR WUu ÿiQJ NӇ7LӅP QăQJ NӃW KӧS KDL Nӻ WKXұW Qj\ WURQJ PҥQJ FҧP ELӃQ VҥF OҥL ÿӇ WăQJ FѭӡQJ WtQK EӅQYӳQJ Yj ÿiQJ WLQ Fұ\ FӫD PҥQJ Oj PӝW KѭӟQJ QJKLrQ FӭX WLӅP QăQJ WURQJ WѭѫQJ ODL 9LӋF

iS GөQJ FiF JLҧL SKiS Qj\ ÿһF ELӋW Oj QKӳQJ WLӃQ Eӝ PӟL WURQJ KӑF WăQJ FѭӡQJ VkX Fy WKӇ

Pӣ ÿѭӡQJ FKR FiF P{ KuQK PҥQJ FҧP ELӃQ KLӋX TXҧ KѫQ JL~S FK~QJ WD JLiP ViW Yj TXҧQ

Oê WӕW KѫQ FiF KӋ WKӕQJ NKiF QKDX WURQJ FXӝF VӕQJ FӫD FK~QJ WD

LLL

-JTU PG 'JHVSFT

 :LUHOHVV VHQVRU QHWZRUN DUFKLWHFWXUH VRXUFH %DKUL          

 3DUHWR GRPLQDQFH 6RXUFH 9HUPD HW DO                 

 ([DPSOHV RI WKH QRQGRPLQDWHG VRUWLQJ DOJRULWKP DQG FURZGLQJ GLVWDQFHFDOFXODWLRQ 6RXUFH 9HUPD HW DO  WR OHDUQ RSWLPDO SROLFLHV LQ HQYLURQPHQWV ZLWK GLVFUHWH DQG VPDOO VWDWHDFWLRQ VSDFHV

'HHS UHLQIRUFHPHQW OHDUQLQJ '5/ LV D UHFHQW H[WHQVLRQ RI 5/ WKDW OHYHUDJHV GHHSQHXUDO QHWZRUNV WR HQDEOH OHDUQLQJ LQ WKH KLJKGLPHQVLRQDO VWDWH DQG DFWLRQ VSDFHV ,Q'5/ WKH DJHQW OHDUQV WR GLUHFWO\ PDS UDZ VHQVRU\ LQSXWV HJ SL[HO YDOXHV LQ LPDJHV WRDFWLRQV ZLWKRXW UHO\LQJ RQ KDQGHQJLQHHUHG IHDWXUHV 7KLV LV DFKLHYHG E\ FRPELQLQJ GHHSQHXUDO QHWZRUNV ZLWK WUDGLWLRQDO 5/ DOJRULWKPV VXFK DV 4OHDUQLQJ DQG SROLF\ JUDGLHQWPHWKRGV $V WKLV WKHVLV XVHV DFWRUFULWLF DOJRULWKPV ZKLFK DUH KLJKO\ EDVHG RQ SROLF\JUDGLHQW PHWKRGV ZH SURYLGH EDFNJURXQG NQRZOHGJH DERXW SROLF\ JUDGLHQW PHWKRGV LQWKH IROORZLQJ VHFWLRQ

 1PMJDZ (SBEJFOU NFUIPET

3ROLF\ JUDGLHQW PHWKRGV DUH D FODVV RI UHLQIRUFHPHQW OHDUQLQJ DOJRULWKPV WKDW DLP WR RSWLPL]H SROLFLHV GLUHFWO\ UDWKHU WKDQ FRPSXWLQJ D YDOXH IXQFWLRQ DQG WKHQ GHULYLQJ D SROLF\IURP LW 7KHVH PHWKRGV KDYH JDLQHG SRSXODULW\ UHFHQWO\ GXH WR WKHLU DELOLW\ WR KDQGOHFRQWLQXRXV DFWLRQ VSDFHV DQG WKHLU VXFFHVV LQ DFKLHYLQJ VWDWHRIWKHDUW UHVXOWV LQ YDULRXVDSSOLFDWLRQV LQFOXGLQJ JDPHSOD\LQJ URERWLFV DQG QDWXUDO ODQJXDJH SURFHVVLQJ

3ROLF\ JUDGLHQW PHWKRGV OHDUQ WKH SROLF\ GLUHFWO\ ZLWK D SDUDPHWHUL]HG IXQFWLRQ ZLWK

UHVSHFW WR θ π(a|x; θ) :H WUDLQ WKH DJHQW WR PD[LPL]H WKH H[SHFWHG UHWXUQ 6SHFLILFDOO\



LQ FRQWLQXRXV VSDFH ZLWK X1DV WKH LQLWLDO VWDUWLQJ VWDWH

J(θ) = V π θ (X1) =Eπ θ [G1|X1]. 

:H PD[LPL]H WKH REMHFWLYH IXQFWLRQ J(θ) XVLQJ WKH JUDGLHQW DVFHQW PHWKRG 7KH JUDGLHQW

RI J FDQ EH FRPSXWHG XVLQJ 3ROLF\ *UDGLHQW 7KHRUHP 6XWWRQ DQG %DUWR  DV IROORZV

∇J(θ) = E π θ[∇ ln π(a|x; θ)Q π (x, a)].  +DYLQJ WKH JUDGLHQW ZH FDQ RSWLPL]H SROLFLHV GLUHFWO\ XVLQJ JUDGLHQW DVFHQW DOJRULWKPV+RZHYHU HDUO\ SROLF\ JUDGLHQW PHWKRGV VXIIHUHG IURP KLJK YDULDQFH ZKLFK PDGH WKHPGLIILFXOW WR FRQYHUJH WR RSWLPDO SROLFLHV

7R DGGUHVV WKLV LVVXH DFWRUFULWLF PHWKRGV FRPELQH D SROLF\ QHWZRUN WKH DFWRU ZLWK DYDOXH IXQFWLRQ QHWZRUN WKH FULWLF WR SURYLGH D PRUH VWDEOH HVWLPDWH RI WKH SROLF\ JUDGLHQW6SHFLILFDOO\ WZR QHWZRUNV DUH PDLQWDLQHG LQ DFWRUFULWLF PHWKRGV 2QH QHWZRUN LV XVHG

IRU OHDUQLQJ YDOXH IXQFWLRQ QDPHO\ WKH FULWLF GHQRWHG V ψ DQG DQRWKHU OHDUQV WKH PDSSLQJ

EHWZHHQ VWDWH DQG DFWLRQV GLUHFWO\ NQRZQ DV WKH DFWRU GHQRWHG π θ 7KH FULWLF LV XVHG WRFULWLFL]H WKH DFWLRQV PDGH E\ WKH DFWRU DQG WKH DFWRU DGMXVWV LWV SDUDPHWHUV LQ WKH GLUHFWLRQVXJJHVWHG E\ WKH FULWLF 7KH JUDGLHQW RI WKH DFWRU LV QRZ JLYHQ E\

LV DFKLHYHG E\ FUHDWLQJ VKRUWFXWV WKDW FRPELQH WKH HQWLUH LQSXW VHTXHQFH LQWR D FRQWH[WYHFWRU ZLWK ZHLJKWV DVVLJQHG WR UHSUHVHQW KRZ PXFK DWWHQWLRQ LV GHYRWHG WR HDFK LQSXW

0DWKHPDWLFDOO\ OHW XV GHQRWH WKH HQFRGHU DQG GHFRGHU KLGGHQ VWDWHV ZLWK (e1, e2, , e n)

DQG (d1, d2, , d n)D FRQWH[W YHFWRU DW GHFRGLQJ WLPH i LV JLYHQ E\

u i j = f (W1e j + W2d i ), j ∈ {1, 2, , n}. 



7KLV FRQWH[W YHFWRU c LV ODWHU FRQFDWHQDWHG ZLWK GHFRGHU VWDWH d WR PDNH D SUHGLFWLRQ

RU FRPSXWH D KLGGHQ YHFWRU IRU WKH QH[W VWHSV RI WKH UHFXUUHQW PRGHO

 1PJOUJOH NFDIBOJTN

7KH SRLQWLQJ PHFKDQLVP LV D WHFKQLTXH ILUVW SURSRVHG LQ 9LQ\DOV HW DO  WR SURGXFHGLVFUHWH RXWSXWV WKDW FRUUHVSRQG WR SRVLWLRQV LQ WKH LQSXW )RU H[DPSOH LQ WKH FRPELQDWRULDO SUREOHP  7UDYHO 6DLOLQJ 3UREOHP WKH VROXWLRQ LV D SHUPXWDWLRQ RI WKH LQSXW SRVLWLRQV,Q 9LQ\DOV HW DO   WKH DXWKRUV SURSRVHG D 3RLQWHU QHWZRUN ZKLFK LV DQ HQFRGHUGHFRGHU /670 PRGHO 7KH LQSXW LQFOXGLQJ D VHTXHQFH RI WKH QRGH¶V SRVLWLRQ LV HQFRGHG

E\ DQ /670 HQFRGHU ,Q WKH GHFRGHU LQVWHDG RI EOHQGLQJ WKH HQFRGHU KLGGHQ VWDWHV e j

LQWR D FRQWH[W YHFWRU c DW HDFK GHFRGHU VWHS D UHGXFWLRQ RI DWWHQWLRQ PHFKDQLVP LV XVHG WR

SRLQW WR D PHPEHU RI WKH LQSXW VHTXHQFH WR EH VHOHFWHG DV WKH RXWSXW

u i j = f(W1e j + W2d i ), j ∈ {1, 2, , n}, 

ZKHUH a i

j LV FRQVLGHUHG DV WKH SUREDELOLW\ WR VHOHFW LQSXW j LQ WKH GHFRGHU VWHS i 7KH QRGH

ZLWK WKH KLJKHVW SUREDELOLW\ LV FKRVHQ WR EH YLVLWHG QH[W 7KH SURFHGXUH LV LWHUDWLYHO\ UHSHDWHG WR REWDLQ WKH ILQDO VROXWLRQ



HW DO   )LJXUH  SURYLGHV D VLPSOLILHG GLDJUDP RI WKH VHQVRU QRGH DUFKLWHFWXUH

)LJXUH  $ EORFN GLDJUDP RI WKH DUFKLWHFWXUH RI WKH VHQVRU QRGH LQ WKH :61

6HQVRU QRGHV DUH JHQHUDOO\ ORZFRVW UHVXOWLQJ LQ D EDWWHU\ ZLWK D ORZ FDSDFLW\ DQGXVXDOO\ QRQUHQHZDEOH 7KHUHIRUH H[WHQGLQJ WKH QHWZRUN¶V OLIHWLPH LV D FUXFLDO IDFWRUWKDW GHWHUPLQHV WKH RYHUDOO HIILFLHQF\ DQG HIIHFWLYHQHVV RI WKHVH QHWZRUNV 7KH QHWZRUNOLIHWLPH LV JHQHUDOO\ GHILQHG DV WKH WLPH XQWLO HLWKHU FRYHUDJH RU FRQQHFWLYLW\ LV ORVW =KDRDQG *XUXVDP\   1XPHURXV HIIRUWV KDYH EHHQ GHGLFDWHG WR H[WHQGLQJ WKH OLIHVSDQ RI

ZLUHOHVV VHQVRU QHWZRUNV :61V  ZKLFK FDQ EH FODVVLILHG LQWR WZR PDLQ JURXSV VHQVRU

IXQFWLRQLQJ RSWLPL]DWLRQ DQG HQHUJ\ UHSOHQLVKPHQW

6HQVRU IXQFWLRQLQJ RSWLPL]DWLRQ IRFXVHV RQ HQKDQFLQJ WKH HIILFLHQF\ RI VHQVRU QRGHVZKLOH UHGXFLQJ WKHLU HQHUJ\ FRQVXPSWLRQ 9DULRXV PHWKRGV KDYH EHHQ SURSRVHG LQ WKLVDUHD VXFK DV GDWD UHGXFWLRQ ZKLFK LQYROYHV UHPRYLQJ UHGXQGDQW RU XQQHFHVVDU\ GDWD WRUHGXFH WKH DPRXQW RI LQIRUPDWLRQ WUDQVPLWWHG DQG FRQVHTXHQWO\ WKH HQHUJ\ FRQVXPSWLRQ *R\DO HW DO