LoRa網(wǎng)絡(luò)中基于深度強(qiáng)化學(xué)習(xí)的信息年齡優(yōu)化

程克非; 陳彩蝶; 羅佳; 陳前斌

doi:10.11999/JEIT240404

LoRa網(wǎng)絡(luò)中基于深度強(qiáng)化學(xué)習(xí)的信息年齡優(yōu)化

doi: 10.11999/JEIT240404 cstr: 32379.14.JEIT240404

程克非¹,
陳彩蝶¹,
羅佳^{1, 2, ,},
陳前斌²

1.
重慶郵電大學(xué)網(wǎng)絡(luò)空間安全與信息法學(xué)院重慶 400065
2.
重慶郵電大學(xué)通信與信息工程學(xué)院移動(dòng)通信技術(shù)重點(diǎn)實(shí)驗(yàn)室重慶 400065

基金項(xiàng)目: 重慶市教委科學(xué)技術(shù)研究項(xiàng)目(KJQN202400643)

詳細(xì)信息

作者簡介:
程克非：男，博士生導(dǎo)師，研究方向?yàn)闊o線通信網(wǎng)絡(luò)、云計(jì)算與大數(shù)據(jù)、嵌入式系統(tǒng)及應(yīng)用、網(wǎng)絡(luò)空間安全等

陳彩蝶：女，碩士生，研究方向?yàn)長oRa物聯(lián)網(wǎng)

羅佳：男，講師，博士，研究方向?yàn)橄乱淮鸁o線通信網(wǎng)絡(luò)、人工智能、區(qū)塊鏈等

陳前斌：男，教授，博士生導(dǎo)師，研究方向?yàn)閭€(gè)人通信、多媒體信息處理與傳輸、異構(gòu)蜂窩網(wǎng)絡(luò)等

通訊作者:
羅佳　s220802003@stu.cqupt.edu.cn

中圖分類號(hào): TN929.5
計(jì)量
- 文章訪問數(shù): 393
- HTML全文瀏覽量: 173
- PDF下載量: 28
- 被引次數(shù): 0
出版歷程
- 收稿日期: 2024-04-04
- 修回日期: 2025-01-08
- 網(wǎng)絡(luò)出版日期: 2025-01-25
- 刊出日期: 2025-02-28

Optimizing Age of Information in LoRa Networks via Deep Reinforcement Learning

1.
School of Cyber Security and Information Law, Chongqing University of Posts and Telecommunications, Chongqing 400065, China
2.
Key Laboratory of Mobile Communication Technology, School of Communication and Information Engineering, Chongqing University of Posts and Telecommunications, Chongqing 400065, China

Funds: The Science and Technology Research Program of Chongqing Municipal Education Commission (KJQN202400643)

摘要

摘要: 信息年齡(AoI)是信息新鮮度的衡量指標(biāo)，針對時(shí)間敏感的物聯(lián)網(wǎng)，最小化AoI顯得尤為重要。該文基于LoRa網(wǎng)絡(luò)的智能交通環(huán)境，分析Slot-Aloha協(xié)議下的AoI優(yōu)化策略，建立了Slot-Aloha協(xié)議下數(shù)據(jù)包之間傳輸碰撞和等待時(shí)間的系統(tǒng)模型。通過分析指出，在LoRa上行傳輸過程中，隨著數(shù)據(jù)包數(shù)量增多，AoI主要受到數(shù)據(jù)包碰撞影響。為克服優(yōu)化問題中動(dòng)作空間過大導(dǎo)致難以實(shí)現(xiàn)有效求解的問題，該文采用連續(xù)動(dòng)作空間映射離散動(dòng)作空間的方式，使用柔性動(dòng)作-評價(jià) (SAC)算法對LoRa網(wǎng)絡(luò)下的AoI進(jìn)行優(yōu)化。仿真結(jié)果顯示，SAC算法優(yōu)于傳統(tǒng)算法與傳統(tǒng)深度強(qiáng)化學(xué)習(xí)算法，可有效降低網(wǎng)絡(luò)的平均AoI。
- 信息年齡 /
- LoRa /
- 柔性動(dòng)作-評價(jià)算法 /
- 深度強(qiáng)化學(xué)習(xí) /
- 優(yōu)化策略
Abstract: Age of Information (AoI) quantifies information freshness, which is critical for time-sensitive Internet of Things (IoT) applications. This paper investigates AoI optimization in an LoRa network under the Slot-Aloha protocol in an intelligent transportation environment. A system model is established to characterize transmission collisions and packet waiting times. Analytical results indicate that in LoRa uplink transmission, as the number of packets increases, AoI is primarily influenced by packet collisions. To address the challenge of a large action space hindering effective solutions, this study maps the continuous action space to a discrete action space and employs the Soft Actor-Critic (SAC) algorithm for AoI optimization. Simulation results demonstrate that the SAC algorithm outperforms conventional algorithms and traditional deep reinforcement learning approaches, effectively reducing the network’s average AoI. Objective With the rapid development of intelligent transportation systems, ensuring the real-time availability and accuracy of traffic data has become essential, particularly in transmission systems for traffic monitoring cameras and related equipment. Long-range, low-power radio frequency (LoRa) networks have emerged as a key technology for sensor connectivity in intelligent transportation due to their advantages of low power consumption, wide coverage, and long-distance communication. However, in urban environments, LoRa networks are prone to frequent data collisions when multiple devices transmit simultaneously, which affects information timeliness and, consequently, the effectiveness of traffic management decisions. This study focuses on optimizing data packet timeliness in LoRa networks to enhance communication efficiency. Specifically, it aims to improve AoI under the Slotted Aloha protocol by analyzing the effects of packet collisions and over-the-air transmission time. Based on this analysis, an optimization method using deep reinforcement learning is proposed, employing the SAC algorithm to minimize AoI. The goal is to achieve lower latency and a higher data transmission success rate in an intelligent transportation environment with frequent data transmissions, thereby improving overall system performance and ensuring real-time information availability to meet the freshness requirements of intelligent transportation systems. Method To address the requirements for information freshness in intelligent transportation scenarios, this study investigates the optimization of packet AoI in LoRa networks under the Slotted Aloha protocol. A system model is established to analyze packet collisions and over-the-air transmission time, providing theoretical support for enhancing information transmission efficiency. Given the Markovian nature of AoI evolution, the optimization problem is formulated as a Markov Decision Process (MDP) and solved using the SAC algorithm in deep reinforcement learning. Results and Discussions The study examines AoI variations during collisions (Fig. 2) and develops a collision model for data packet transmission (Fig. 4). Simulation results indicate that the SAC algorithm outperforms the Temporal Difference (TD) algorithm and conventional methods (Fig. 6). As the number of terminals increases, the system’s average AoI also increases (Fig. 7). Additionally, the variations in average AoI under different time slots for the SAC and TD3 algorithms are analyzed (Fig. 8). Conclusions Given the limited research on AoI in LoRa networks, this study examines the AoI optimization problem in LoRa uplink packet transmission within an intelligent traffic management environment and proposes a packet collision model under the Slotted Aloha protocol. The greedy algorithm and SAC algorithm are employed for AoI optimization. Simulation results demonstrate that the greedy algorithm outperforms conventional deep reinforcement learning algorithms but remains less effective than the SAC algorithm. The SAC algorithm significantly improves AoI optimization in LoRa networks. However, this study focuses solely on AoI optimization without considering energy consumption and packet loss rate. Future research should explore the trade-offs between energy efficiency, packet loss, and AoI optimization to minimize energy consumption and data loss. Additionally, this study does not address heterogeneous network scenarios. In environments where LoRa networks coexist with other communication technologies (e.g., Wi-Fi, Bluetooth, NB-IoT), challenges related to interoperability, data consistency, and network management arise. Investigating AoI optimization in heterogeneous transmission environments could further enhance the performance and reliability of LoRa networks in complex applications such as intelligent traffic management.
- Age of Information (AoI) /
- LoRa /
- Soft Actor-Critic algorithm (SAC) /
- Deep reinforcement learning /
- Optimization strategy

HTML全文

圖 1 系統(tǒng)模型

下載: 全尺寸圖片幻燈片

圖 2 基于時(shí)隙Aloha的數(shù)據(jù)包AoI變化情況

下載: 全尺寸圖片幻燈片

圖 3 數(shù)據(jù)包傳輸情況

下載: 全尺寸圖片幻燈片

圖 4 各終端數(shù)據(jù)包傳輸情況

下載: 全尺寸圖片幻燈片

圖 5 狀態(tài)轉(zhuǎn)移圖

下載: 全尺寸圖片幻燈片

圖 6 不同算法的收斂曲線

下載: 全尺寸圖片幻燈片

圖 7 不同終端數(shù)量下的AoI

下載: 全尺寸圖片幻燈片

圖 8 TD3算法和SAC算法在不同時(shí)隙長度$ {T}_{\mathrm{s}\mathrm{l}} $下平均AoI變化

下載: 全尺寸圖片幻燈片

表 1 空中傳輸時(shí)間

SF	7	8	9	10	11	12
$ {T}^{\mathrm{a}} $ (ms)	73.1	128	227.6	409.6	744.7	1365.3

下載: 導(dǎo)出CSV

1 貪婪算法

輸入：終端集合$ \mathrm{s}\mathrm{e}\mathrm{n}\mathrm{s}\mathrm{o}\mathrm{r}\mathrm{s}=\{{\mathrm{s}\mathrm{e}\mathrm{n}}_{1},{\mathrm{s}\mathrm{e}\mathrm{n}}_{2},\cdots ,{\mathrm{s}\mathrm{e}\mathrm{n}}_{n}\} $
輸出：每個(gè)終端分配的SF和信道集合
$ \mathrm{r}\mathrm{e}\mathrm{s}\mathrm{u}\mathrm{l}\mathrm{t}=\{\left({\mathrm{S}\mathrm{F}}_{1},{C}_{1}\right),\left({\mathrm{S}\mathrm{F}}_{2},{C}_{2}\right),\cdots ,({\mathrm{S}\mathrm{F}}_{n},{C}_{n}\left)\right\} $
(1) $ \mathrm{r}\mathrm{e}\mathrm{s}\mathrm{u}\mathrm{l}\mathrm{t} \leftarrow \mathrm{\varnothing },\mathrm{s}\mathrm{f}\in \left\{\mathrm{7,8},\cdots ,12\right\},\mathrm{信}\mathrm{道}C\in \{\mathrm{0,1},\cdots, c\} $
(2) $ \bf{f}\bf{o}\bf{r}\;i=\mathrm{1,2},\cdots ,N\;\bfq7j3ldu95\bf{o} $
(3) 　$ {\mathrm{S}\mathrm{F}}_{i}\leftarrow \mathrm{隨}\mathrm{機(jī)}\mathrm{從}{\bf{s}\bf{f}}\mathrm{集}\mathrm{合}\mathrm{中}\mathrm{選}\mathrm{擇} $
(4) 　$ {{C}}_{i}\leftarrow \mathrm{隨}\mathrm{機(jī)}\mathrm{從}C\mathrm{集}\mathrm{合}\mathrm{中}\mathrm{選}\mathrm{擇} $
(5) 　$ \mathrm{r}\mathrm{e}\mathrm{s}\mathrm{u}\mathrm{l}\mathrm{t}\leftarrow \mathrm{r}\mathrm{e}\mathrm{s}\mathrm{u}\mathrm{l}\mathrm{t}\cup ({\mathrm{S}\mathrm{F}}_{i},{C}_{i}) $
(6) $ \bf{e}\bf{n}\bfq7j3ldu95\;\bf{f}\bf{o}\bf{r} $
(7) $ \bf{f}\bf{o}\bf{r}\;j=\mathrm{1,2},\cdots ,N\;\bfq7j3ldu95\bf{o} $
(8) 　記錄第$ j $時(shí)隙下最優(yōu)的SF $ {\mathrm\mathrm{e}\mathrm{s}\mathrm{t}{\mathrm{SF}}}_{j}=0 $
(9) 　記錄第$ j $時(shí)隙下最優(yōu)的信道C $ {\mathrm\mathrm{e}\mathrm{s}\mathrm{t}C}_{j}=0 $
(10) 　$ \mathrm{m}\mathrm{i}\mathrm{n}\mathrm{ }\mathrm{ }\mathrm{ }\mathrm{ }\mathrm{ }\mathrm{ }\mathrm{a}\mathrm{v}\mathrm{g}\mathrm{A}\mathrm{o}\mathrm{I}=\mathrm{M}\_\mathrm{I}\mathrm{N}\mathrm{T} $
(11) 　$ \bf{f}\bf{o}\bf{r}\;\mathrm{s}\mathrm{f}\mathrm{ }\leftarrow \mathrm{ }\mathrm{ }\mathrm{ }\mathrm{ }\mathrm{ }\mathrm{ }7\;\mathrm{t}\mathrm{o}\;12\;\bfq7j3ldu95\bf{o} $
(12) 　　$ \bf{f}\bf{o}\bf{r}\;C\leftarrow 0\;\mathrm{t}\mathrm{o}\;c\;\bfq7j3ldu95\bf{o} $
(13) 　　$ \mathrm{a}\mathrm{v}\mathrm{g}\mathrm{A}\mathrm{o}\mathrm{I}=\mathrm{a}\mathrm{v}\mathrm{e}\mathrm{r}\mathrm{a}\mathrm{g}\mathrm{e}\left(\mathrm{A}\mathrm{o}\mathrm{I}\right) $
(14) 　　$ \text{if avgAoI < minavgAoI} $
(15) 　　　$ \text{minavgAoI=avgAoI} $
(16) 　　　$ {\mathrm\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{S}\mathrm{F}}_{j}=\mathrm{s}\mathrm{f} $
(17) 　　　$ {\mathrm\mathrm{e}\mathrm{s}\mathrm{t}C}_{j}=C $
(18) 　　$ \bf{end}\;\bf{f}\bf{o}\bf{r} $
(19) 　　$ \bf{e}\bf{n}\bfq7j3ldu95\;\bf{f}\bf{o}\bf{r} $
(20) 　$ \mathrm{r}\mathrm{e}\mathrm{s}\mathrm{u}\mathrm{l}\mathrm{t}\leftarrow \mathrm{r}\mathrm{e}\mathrm{s}\mathrm{u}\mathrm{l}\mathrm{t}\cup ({\mathrm\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{S}\mathrm{F}}_{i},{\mathrm\mathrm{e}\mathrm{s}\mathrm{t}C}_{i}) $
(21) 　$ \bf{e}\bf{n}\bfq7j3ldu95\;\bf{f}\bf{o}\bf{r} $
(22) $ \bf{r}\bf{e}\bf{t}\bf{u}\bf{r}\bf{n}\;\bf{r}\bf{e}\bf{s}\bf{u}\bf{l}\bf{t} $

下載: 導(dǎo)出CSV

表 2 實(shí)驗(yàn)參數(shù)值

參數(shù)名	值
信道數(shù)量($ c $)	2
終端數(shù)量($ N $)	12
SF數(shù)量	6
編碼率($ \mathrm{C}\mathrm{R} $)	4/5
帶寬($ \mathrm{B}\mathrm{W} $)	125 kHz
數(shù)據(jù)包大小($ {L}_{\mathrmq7j3ldu95} $)	50 Byte
step總數(shù)($ {T}_{\mathrm{s}\mathrm{t}} $)	500
時(shí)隙長度($ {T}_{\mathrm{s}\mathrm{l}} $)	500 ms

下載: 導(dǎo)出CSV

參考文獻(xiàn)(21)

[1]	KAUL S, YATES R, and GRUTESER M. Real-time status: How often should one update?[C]. IEEE INFOCOM, Orlando, USA, 2012: 2731–2735. doi: 10.1109/INFCOM.2012.6195689.
[2]	INOUE Y, MASUYAMA H, TAKINE T, et al. A general formula for the stationary distribution of the age of information and its application to single-server queues[J]. IEEE Transactions on Information Theory, 2019, 65(12): 8305–8324. doi: 10.1109/TIT.2019.2938171.
[3]	BEDEWY A M, SUN Yin, and SHROFF N B. Minimizing the age of information through queues[J]. IEEE Transactions on Information Theory, 2019, 65(8): 5215–5232. doi: 10.1109/TIT.2019.2912159.
[4]	HE Qing, YUAN Di, and EPHREMIDES A. Optimal link scheduling for age minimization in wireless systems[J]. IEEE Transactions on Information Theory, 2018, 64(7): 5381–5394. doi: 10.1109/TIT.2017.2746751.
[5]	KADOTA I, SINHA A, UYSAL-BIYIKOGLU E, et al. Scheduling policies for minimizing age of information in broadcast wireless networks[J]. IEEE/ACM Transactions on Networking, 2018, 26(6): 2637–2650. doi: 10.1109/TNET.2018.2873606.
[6]	WU Beining, CAI Zhengkun, WU Wei, et al. AoI-aware resource management for smart health via deep reinforcement learning[J]. IEEE Access, 2023, 11: 81180–81195. doi: 10.1109/ACCESS.2023.3299340.
[7]	PENG Kai, XIAO Peiyun, WANG Shangguang, et al. AoI-aware partial computation offloading in IIoT with edge computing: A deep reinforcement learning based approach[J]. IEEE Transactions on Cloud Computing, 2023, 11(4): 3766–3777. doi: 10.1109/TCC.2023.3328614.
[8]	WANG Hao, LIU Chi, YANG Haoming, et al. Ensuring threshold AoI for UAV-assisted mobile crowdsensing by multi-agent deep reinforcement learning with transformer[J]. IEEE/ACM Transactions on Networking, 2024, 32(1): 566–581. doi: 10.1109/TNET.2023.3289172.
[9]	WANG Zhuoyao, XU Xiaokang, and ZHAO Jin. Spreading factor allocation and rate adaption for minimizing age of information in LoRaWAN[C]. 2022 IEEE 24th International Conference on High Performance Computing & Communications; 8th International Conference on Data Science & Systems; 20th International Conference on Smart City; 8th International Conference on Dependability in Sensor, Cloud & Big Data Systems & Application (HPCC/DSS/SmartCity/DependSys), Hainan, China, 2022: 482–489. doi: 10.1109/HPCC-DSS-SmartCity-DependSys57074.2022.00092.
[10]	CUOMO F, CAMPO M, CAPONI A, et al. EXPLoRa: Extending the performance of LoRa by suitable spreading factor allocations[C]. Proceedings of the 13th International Conference on Wireless and Mobile Computing, Networking and Communications, Rome, Italy, 2017: 1–8. doi: 10.1109/WiMOB.2017.8115779.
[11]	GEORGIOU O and RAZA U. Low power wide area network analysis: Can LoRa scale?[J]. IEEE Wireless Communications Letters, 2017, 6(2): 162–165. doi: 10.1109/LWC.2016.2647247.
[12]	HAMDI R, QARAQE M, and ALTHUNIBAT S. Dynamic spreading factor assignment in LoRa wireless networks[C]. 2020 IEEE International Conference on Communications (ICC), Dublin, Ireland, 2020: 1–5. doi: 10.1109/ICC40277.2020.9149243.
[13]	FARHAD A, KIM D H, STHAPIT P, et al. Interference-aware spreading factor assignment scheme for the massive LoRaWAN network[C]. International Conference on Electronics, Information, and Communications, Auckland, New Zealand, 2019: 1–2. doi: 10.23919/ELINFOCOM.2019.8706416.
[14]	FARHAD A, KIM D H, and PYUN J Y. Resource allocation to massive internet of things in LoRaWANs[J]. Sensors, 2020, 20(9): 2645. doi: 10.3390/s20092645.
[15]	BELTRAMELLI L, MAHMOOD A, ?STERBERG P, et al. LoRa beyond aloha: An investigation of alternative random access protocols[J]. IEEE Transactions on Industrial Informatics, 2021, 17(5): 3544–3554. doi: 10.1109/TII.2020.2977046.
[16]	WANG Jiwen, YU Jihong, CHEN Xiaoming, et al. Age of information for frame slotted aloha[J]. IEEE Transactions on Communications, 2023, 71(4): 2121–2135. doi: 10.1109/TCOMM.2023.3244214.
[17]	CHEIKH I, SABIR E, AOUAMI R, et al. Throughput-delay tradeoffs for slotted-aloha-based LoRaWAN networks[C]. International Conference on Wireless Communications and Mobile Computing, Harbin, China, 2021: 2020–2025. doi: 10.1109/IWCMC51323.2021.9498969.
[18]	HAMDI R, BACCOUR E, ERBAD A, et al. LoRa-RL: Deep reinforcement learning for resource management in hybrid energy LoRa wireless networks[J]. IEEE Internet of Things Journal, 2022, 9(9): 6458–6476. doi: 10.1109/JIOT.2021.3110996.
[19]	HONG Shengguang, YAO Fang, ZHANG Fengyun, et al. Reinforcement learning approach for SF allocation in LoRa network[J]. IEEE Internet of Things Journal, 2023, 10(20): 18259–18272. doi: 10.1109/JIOT.2023.3279429.
[20]	WARET A, KANEKO M, GUITTON A, et al. LoRa throughput analysis with imperfect spreading factor orthogonality[J]. IEEE Wireless Communications Letters, 2019, 8(2): 408–411. doi: 10.1109/LWC.2018.2873705.
[21]	HAARNOJA T, ZHOU A, ABBEEL P, et al. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor[C]. The 35th International Conference on Machine Learning, Stockholm, Sweden, 2018: 1856–1865.