一種車(chē)載服務(wù)的快速深度Q學(xué)習(xí)網(wǎng)絡(luò)邊云遷移策略
doi: 10.11999/JEIT190612 cstr: 32379.14.JEIT190612
-
1.
中南大學(xué)計(jì)算機(jī)學(xué)院 長(zhǎng)沙 410083
-
2.
中南大學(xué)自動(dòng)化學(xué)院 長(zhǎng)沙 410083
基金項(xiàng)目: 國(guó)家自然科學(xué)基金(61873353, 61672539)
A Fast Deep Q-learning Network Edge Cloud Migration Strategy for Vehicular Service
-
1.
School of Computer Science and Engineering, Central South University, Changsha 410083, China
-
2.
School of Automation, Central South University, Changsha 410083, China
Funds: The National Natural Science Foundation of China(61873353, 61672539)
-
摘要: 智能網(wǎng)聯(lián)交通系統(tǒng)中車(chē)載用戶的高速移動(dòng)，不可避免地造成了數(shù)據(jù)在邊緣服務(wù)器之間頻繁遷移，產(chǎn)生了額外的通信回傳時(shí)延，對(duì)邊緣服務(wù)器的實(shí)時(shí)計(jì)算服務(wù)帶來(lái)了巨大的挑戰(zhàn)。為此，該文提出一種基于車(chē)輛運(yùn)動(dòng)軌跡的快速深度Q學(xué)習(xí)網(wǎng)絡(luò)(DQN-TP)邊云遷移策略，實(shí)現(xiàn)數(shù)據(jù)遷移的離線評(píng)估和在線決策。車(chē)載決策神經(jīng)網(wǎng)絡(luò)實(shí)時(shí)獲取接入的邊緣服務(wù)器網(wǎng)絡(luò)狀態(tài)和通信回傳時(shí)延，根據(jù)車(chē)輛的運(yùn)動(dòng)軌跡進(jìn)行虛擬機(jī)或任務(wù)遷移的決策，同時(shí)將實(shí)時(shí)的決策信息和獲取的邊緣服務(wù)器網(wǎng)絡(luò)狀態(tài)信息發(fā)送到云端的經(jīng)驗(yàn)回放池中；評(píng)估神經(jīng)網(wǎng)絡(luò)在云端讀取經(jīng)驗(yàn)回放池中的相關(guān)信息進(jìn)行網(wǎng)絡(luò)參數(shù)的優(yōu)化訓(xùn)練，定時(shí)更新車(chē)載決策神經(jīng)網(wǎng)絡(luò)的權(quán)值，實(shí)現(xiàn)在線決策的優(yōu)化。最后仿真驗(yàn)證了所提算法與虛擬機(jī)遷移算法和任務(wù)遷移算法相比能有效地降低時(shí)延。Abstract: The high-speed movement of vehicles inevitably leads to frequent data migration between edge servers and increases communication delay, which brings great challenges to the real-time computing service of edge servers. To solve this problem, a real-time reinforcement learning method based on Deep Q-learning Networks according to vehicle motion Trajectory Process (DQN-TP) is proposed. The proposed algorithm separates the decision-making process from the training process by using two neural networks. The decision neural network obtains the network state in real time according to the vehicle’s movement track and chooses the migration method in the virtual machine migration and task migration. At the same time, the decision neural network uploads the decision records to the memory replay pool in the cloud. The evaluation neural network in the cloud trains with the records in the memory replay pool and periodically updates the parameters to the on-board decision neural network. In this way, training and decision-making can be carried out simultaneously. At last, a large number of simulation experiments show that the proposed algorithm can effectively reduce the latency compared with the existing methods of task migration and virtual machine migration.
-
表 1 變量表
變量名 變量符號(hào) 決策周期長(zhǎng)度 $\sigma $ 決策周期 $t$ 邊緣服務(wù)器數(shù)量 $i$ 車(chē)載用戶位置 ${\nu _t}$ 邊緣服務(wù)器位置 ${\mu _m}$ 路徑損失參數(shù) $\delta $ 路旁單元覆蓋半徑 $r$ 任務(wù)大小 ${q_{\rm s}}$ 任務(wù)最大容忍時(shí)延 ${q_{\rm d}}$ 傳輸功率 ${P_{\rm s}}$ 時(shí)延 $T$ 虛擬機(jī)所在位置 ${ D}$ 
下載: 導(dǎo)出CSV
表 2 DQN-TP算法
算法1: DQN-TP算法 (1) Repeat: (2) 車(chē)載用戶上傳車(chē)載決策神經(jīng)網(wǎng)絡(luò)的經(jīng)驗(yàn)$({X_t},{a_t},{U_t},{X_{t + 1}})$到經(jīng)驗(yàn)回放池； (3) While $t \ne $最后一個(gè)周期do (4) 從經(jīng)驗(yàn)回放池中隨機(jī)抽取$n$個(gè)經(jīng)驗(yàn)作為一個(gè)mini-batch； (5) 將${X_t},{a_t}$作為評(píng)估神經(jīng)網(wǎng)絡(luò)的輸入獲得${Q_{\pi} }({X_t},{a_t};\theta )$，將${X_{t + 1}}$作為決策神經(jīng)網(wǎng)絡(luò)的輸入獲得${Q_{\pi } }({X_{t + 1} },{a_{t + 1} };{\theta ^-})$； (6) 根據(jù)式(13)、式(14)訓(xùn)練神經(jīng)網(wǎng)絡(luò)； (7) End While (8) 每訓(xùn)練$c$次將云端的神經(jīng)網(wǎng)絡(luò)參數(shù)更新給車(chē)載神經(jīng)網(wǎng)絡(luò)$\theta \to {\theta ^{^\_}}$； (9) 車(chē)載用戶使用$\varepsilon {\rm{ - }}$貪婪算法選擇動(dòng)作-狀態(tài)值函數(shù)最高的動(dòng)作作為車(chē)載用戶動(dòng)作執(zhí)行； (10) End
下載: 導(dǎo)出CSV
表 3 仿真參數(shù)設(shè)定
參數(shù)名 參數(shù)符號(hào) 參數(shù)值 決策周期 $\sigma $ 10–3 s 邊緣服務(wù)器數(shù)量 $i$ 10 路徑損失參數(shù) $\delta $ 1.5 帶寬 $W$ 4 MHz 路旁單元覆蓋半徑 $r$ 500 m 效用函數(shù)參數(shù) $k$ 1.3 效用函數(shù)參數(shù) $b$ 0.1 記憶回放池最大存儲(chǔ)數(shù) $o$ 3000 Mini-batch大小 $n$ 500 參數(shù)更新間隔步長(zhǎng) $c$ 80 神經(jīng)網(wǎng)絡(luò)層數(shù) 無(wú) 4 神經(jīng)元總數(shù) 無(wú) 100
下載: 導(dǎo)出CSV
-
ZHU Li, YU F R, WANG Yige, et al. Big data analytics in intelligent transportation systems: A survey[J]. IEEE Transactions on Intelligent Transportation Systems, 2019, 20(1): 383–398. doi: 10.1109/TITS.2018.2815678 D’OREY P M and FERREIRA M. ITS for sustainable mobility: A survey on applications and impact assessment tools[J]. IEEE Transactions on Intelligent Transportation Systems, 2014, 15(2): 477–493. doi: 10.1109/TITS.2013.2287257 彭軍, 馬東, 劉凱陽(yáng), 等. 基于LTE D2D技術(shù)的車(chē)聯(lián)網(wǎng)通信架構(gòu)與數(shù)據(jù)分發(fā)策略研究[J]. 通信學(xué)報(bào), 2016, 37(7): 62–70. doi: 10.11959/j.issn.1000-436x.2016134PENG Jun, MA Dong, LIU Kaiyang, et al. LTE D2D based vehicle networking communication architecture and data distributing strategy[J]. Journal on Communications, 2016, 37(7): 62–70. doi: 10.11959/j.issn.1000-436x.2016134 GAO Kai, HAN Farong, DONG Pingping, et al. Connected vehicle as a mobile sensor for real time queue length at signalized intersections[J]. Sensors, 2019, 19(9): 2059. doi: 10.3390/s19092059 KONG Yue, ZHANG Yikun, WANG Yichuan, et al. Energy saving strategy for task migration based on genetic algorithm[C]. 2018 International Conference on Networking and Network Applications, Xi’an, China, 2018: 330–336. CHEN Xianfu, ZHANG Honggang, WU C, et al. Optimized computation offloading performance in virtual edge computing systems via deep reinforcement learning[J]. IEEE Internet of Things Journal, 2019, 6(3): 4005–4018. doi: 10.1109/JIOT.2018.2876279 SAHA S and HASAN M S. Effective task migration to reduce execution time in mobile cloud computing[C]. The 23rd International Conference on Automation and Computing, Huddersfield, UK, 2017: 1–5. GON?ALVES D, VELASQUEZ K, CURADO M, et al. Proactive virtual machine migration in fog environments[C]. 2018 IEEE Symposium on Computers and Communications, Natal, Brazil, 2018: 742–745. KIKUCHI J, WU C, JI Yusheng, et al. Mobile edge computing based VM migration for QoS improvement[C]. The 6th IEEE Global Conference on Consumer Electronics, Nagoya, Japan, 2017: 1–5. CHOWDHURY M, STEINBACH E, KELLERER W, et al. Context-Aware task migration for HART-Centric collaboration over FiWi based tactile internet infrastructures[J]. IEEE Transactions on Parallel and Distributed Systems, 2018, 29(6): 1231–1246. doi: 10.1109/TPDS.2018.2791406 LU Wei, MENG Xianyu, and GUO Guanfei. Fast service migration method based on virtual machine technology for MEC[J]. IEEE Internet of Things Journal, 2019, 6(3): 4344–4354. doi: 10.1109/JIOT.2018.2884519 WANG Yanting, SHENG Min, WANG Xijun, et al. Mobile-edge computing: Partial computation offloading using dynamic voltage scaling[J]. IEEE Transactions on Communications, 2016, 64(10): 4268–4282. doi: 10.1109/TCOMM.2016.2599530 SUTTON R S and BARTO A G. Reinforcement Learning: An Introduction[M]. Cambridge: MIT Press, 1998: 25–42. SNIA trace data[EB/OL]. http://iotta.snia.org/traces, 2018. -
圖(3) / 表(3)
計(jì)量
- 文章訪問(wèn)數(shù): 2076
- HTML全文瀏覽量: 712
- PDF下載量: 99
- 被引次數(shù): 0
下載:
