基于自適應梯度壓縮的高效聯(lián)邦學習通信機制研究

唐倫; 汪智平; 蒲昊; 吳壯; 陳前斌

doi:10.11999/JEIT211262

基于自適應梯度壓縮的高效聯(lián)邦學習通信機制研究

doi: 10.11999/JEIT211262 cstr: 32379.14.JEIT211262

1.
重慶郵電大學通信與信息工程學院重慶 400065
2.
重慶郵電大學移動通信技術重點實驗室重慶 400065

基金項目: 國家自然科學基金(62071078), 重慶市教委科學技術研究項目(KJZD-M201800601), 川渝聯(lián)合實施重點研發(fā)項目(2021YFQ0053)

詳細信息

作者簡介:
唐倫：男，教授，博士，研究方向為下一代無線通信網(wǎng)絡、異構蜂窩網(wǎng)絡、軟件定義網(wǎng)絡等

汪智平：男，碩士生，研究方向為邊緣智能計算協(xié)同機理、聯(lián)邦學習通信優(yōu)化等

蒲昊：男，碩士生，研究方向為邊緣智能計算資源分配與協(xié)同機理等

吳壯：男，碩士生，研究方向為邊緣智能計算資源分配、無人機動態(tài)規(guī)劃等

陳前斌：男，教授，博士生導師，研究方向為個人通信、多媒體信息處理與傳輸、異構蜂窩網(wǎng)絡等

通訊作者:
汪智平　2609116705@qq.com

中圖分類號: TN929.5
計量
- 文章訪問數(shù): 1618
- HTML全文瀏覽量: 1556
- PDF下載量: 360
- 被引次數(shù): 0
出版歷程
- 收稿日期: 2021-11-12
- 修回日期: 2022-04-22
- 網(wǎng)絡出版日期: 2022-04-28
- 刊出日期: 2023-01-17

Research on Efficient Federated Learning Communication Mechanism Based on Adaptive Gradient Compression

1.
School of Communication and Information Engineering, Chongqing University of Posts and Telecommunications, Chongqing 400065, China
2.
Key Laboratory of Mobile Communications Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, China

Funds: The National Natural Science Foundation of China (62071078), The Science and Technology Research Program of Chongqing Municipal Education Commission (KJZD-M201800601), Sichuan and Chongqing Key R&D Projects (2021YFQ0053)

摘要

摘要: 針對物聯(lián)網(wǎng)(IoTs)場景下，聯(lián)邦學習(FL)過程中大量設備節(jié)點之間因冗余的梯度交互通信而帶來的不可忽視的通信成本問題，該文提出一種閾值自適應的梯度通信壓縮機制。首先，引用了一種基于邊緣-聯(lián)邦學習的高效通信(CE-EDFL)機制，其中邊緣服務器作為中介設備執(zhí)行設備端的本地模型聚合，云端執(zhí)行邊緣服務器模型聚合及新參數(shù)下發(fā)。其次，為進一步降低聯(lián)邦學習檢測時的通信開銷，提出一種閾值自適應的梯度壓縮機制(ALAG)，通過對本地模型梯度參數(shù)壓縮，減少設備端與邊緣服務器之間的冗余通信。實驗結果表明，所提算法能夠在大規(guī)模物聯(lián)網(wǎng)設備場景下，在保障深度學習任務完成準確率的同時，通過降低梯度交互通信次數(shù)，有效地提升了模型整體通信效率。
- 聯(lián)邦學習 /
- 邊緣計算 /
- 通信優(yōu)化 /
- 梯度壓縮
Abstract: Considering the non-negligible communication cost problem caused by redundant gradient interactive communication between a large number of device nodes in the Federated Learning(FL) process in the Internet of Things (IoTs) scenario, gradient communication compression mechanism with adaptive threshold is proposed. Firstly, a structure of Communication-Efficient EDge-Federated Learning (CE-EDFL) is used to prevent device-side data privacy leakage. The edge server acts as an intermediary device to perform device-side local model aggregation, and the cloud performs edge server model aggregation and new parameter delivery. Secondly, in order to reduce further the communication overhead during federated learning detection, a threshold Adaptive Lazily Aggregated Gradient (ALAG) is proposed, which reduces the redundant communication between the device end and the edge server by compressing the gradient parameters of the local model. The experimental results show that the proposed algorithm can effectively improve the overall communication efficiency of the model by reducing the number of gradient interactions while ensuring the accuracy of deep learning tasks in the large-scale IoT device scenario.
- Federated Learning(FL) /
- Edge computing /
- Communication optimization /
- Gradient compression

HTML全文

圖 1 基于邊緣-聯(lián)邦學習的高效通信檢測模型

下載: 全尺寸圖片幻燈片

圖 2 閾值自適應選擇機制

下載: 全尺寸圖片幻燈片

圖 3 不同α取值下的CCI值

下載: 全尺寸圖片幻燈片

圖 4 不同客戶端數(shù)量下模型性能對比

下載: 全尺寸圖片幻燈片

圖 5 4種模型訓練損失對比

下載: 全尺寸圖片幻燈片

圖 6 4種模型檢測精準度對比

下載: 全尺寸圖片幻燈片

圖 7 4種模型全局所需通信次數(shù)

下載: 全尺寸圖片幻燈片

算法1　基于邊緣-聯(lián)邦學習的高效通信算法
輸入：云端初始化參數(shù)$ {\omega _0} $，客戶端數(shù)量N，邊緣設備L
輸出：全局模型參數(shù)$ \omega (k) $
(1) for $ k = 1,2, \cdots ,K $ do
(2) 　　for each Client $ i = 1,2, \cdots ,N $ in parallel do
(3) 　　　使用式(3)計算本地更新梯度$ \omega _i^l(k) $
(4) 　　　end for
(5) 　　if $ k\|{K_1} = 0 $ then
(6) 　　　　for each Edge server $ l = 1,2, \cdots ,L $ in parallel do
(7) 　　　　　使用式(4)計算參數(shù)$ {\omega ^l}(k) $
(8) 　　　　　if $ k\|{K_1}{K_2} \ne 0 $ then
(9) 　　　　　該邊緣端下所有設備參數(shù)保持不變：　　　　　　　　$ {\omega ^l}(k) \leftarrow \omega _i^l(k) $
(10) 　　　　　end if
(11) 　　　　end for
(12) 　　end if
(13) 　　if $ k\|{K_1}{K_2} = 0 $ then
(14) 　　　　使用式(5)計算參數(shù)$ \omega (k) $
(15) 　　　　for each Client $ i = 1,2, \cdots ,N $ in parallel do
(16) 　　　　設備端參數(shù)更新為云端參數(shù)：$ \omega (k) \leftarrow \omega _i^l(k) $
(17) 　　　　end for
(18) 　　end if
(19) end for

下載: 導出CSV

算法2　一種閾值自適應的梯度壓縮算法
輸入：設備端節(jié)點m當前所處迭代k，總迭代次數(shù)K，初始化全局　　　　梯度$ \nabla F $
輸出：完成訓練并符合模型要求的設備節(jié)點$ {M_{\text{L}}} $，M為設備節(jié)點　　　　集合
(1) 初始化全局下發(fā)參數(shù)$ \omega (k - 1) $
(2) 　for $ k = 1,2, \cdots ,K $
(3) 　　　for $ m = 1,2, \cdots ,M $ do
(4) 　　　計算當前m節(jié)點下的本地參數(shù)梯度$ \nabla {F_m}(\theta (k - 1)) $
(5) 　　　判斷參數(shù)梯度是否滿足梯度自檢式(16)
(6) 　　　滿足則跳過本輪通信，本地梯度累計
(7) 　　　參數(shù)梯度更新：$ \nabla {F_m}(\theta (k)) \leftarrow \nabla {F_m}(\theta (k - 1)) $
(8) 　　　不滿足上傳參數(shù)梯度$ \nabla {F_m}(\theta (k - 1)) $至邊緣服務器端
(9) 　　　end for
(10) 　end for

下載: 導出CSV

表 1 不同$ \alpha $取值下的模型檢測準確率及壓縮率

$\alpha $	壓縮前平均通信次數(shù)	壓縮后平均通信次數(shù)	模型測試平均準確率	壓縮率(%)
0.1	400	32	0.9175	8.00
0.2	400	258	0.9298	64.50
0.3	400	270	0.9301	67.50
0.4	400	295	0.9314	73.75
0.5	400	328	0.9335	82.00
0.6	400	342	0.9341	85.50
0.7	400	351	0.9336	87.75
0.8	400	365	0.9352	91.25
0.9	400	374	0.9351	93.75
1.0	400	400	0.9349	100.00

下載: 導出CSV

表 2 不同α, β下各算法性能對比

實驗驗證指標	LAG	EAFLM	ALAG
Acc(Train set)	0.8890	0.9368	0.9342
CR (%)	5.1100	8.7700	8.0000
CCI($ {\beta _1} = 0.4,{\beta _2} = 0.6 $)	0.9274	0.9206	0.9318
CCI($ {\beta _1} = 0.5,{\beta _2} = 0.5 $)	0.9220	0.9226	0.9331
CCI($ {\beta _1} = 0.6,{\beta _2} = 0.4 $)	0.9167	0.9247	0.9315

下載: 導出CSV

參考文獻(14)

[1]	LI Tian, SAHU A K, TALWALKAR A, et al. Federated learning: Challenges, methods, and future directions[J]. IEEE Signal Processing Magazine, 2020, 37(3): 50–60. doi: 10.1109/MSP.2020.2975749
[2]	LUO Siqi, CHEN Xu, WU Qiong, et al. HFEL: Joint edge association and resource allocation for cost-efficient hierarchical federated edge learning[J]. IEEE Transactions on Wireless Communications, 2020, 19(10): 6535–6548. doi: 10.1109/TWC.2020.3003744
[3]	HUANG Liang, FENG Xu, FENG Anqi, et al. Distributed deep learning-based offloading for mobile edge computing networks[J]. Mobile Networks and Applications, 2022, 27: 1123–1130. doi: 10.1007/s11036-018-1177-x
[4]	趙英, 王麗寶, 陳駿君, 等. 基于聯(lián)邦學習的網(wǎng)絡異常檢測[J]. 北京化工大學學報:自然科學版, 2021, 48(2): 92–99. doi: 10.13543/j.bhxbzr.2021.02.012 ZHAO Ying, WANG Libao, CHEN Junjun, et al. Network anomaly detection based on federated learning[J]. Journal of Beijing University of Chemical Technology:Natural Science, 2021, 48(2): 92–99. doi: 10.13543/j.bhxbzr.2021.02.012
[5]	周傳鑫, 孫奕, 汪德剛, 等. 聯(lián)邦學習研究綜述[J]. 網(wǎng)絡與信息安全學報, 2021, 7(5): 77–92. doi: 10.11959/j.issn.2096-109x.2021056 ZHOU Chuanxin, SUN Yi, WANG Degang, et al. Survey of federated learning research[J]. Chinese Journal of Network and Information Security, 2021, 7(5): 77–92. doi: 10.11959/j.issn.2096-109x.2021056
[6]	SHI Weisong, CAO Jie, ZHANG Quan, et al. Edge computing: Vision and challenges[J]. IEEE Internet of Things Journal, 2016, 3(5): 637–646. doi: 10.1109/JIOT.2016.2579198
[7]	WANG Shiqiang, TUOR T, SALONIDIS T, et al. Adaptive federated learning in resource constrained edge computing systems[J]. IEEE Journal on Selected Areas in Communications, 2019, 37(6): 1205–1221. doi: 10.1109/JSAC.2019.2904348
[8]	ABESHU A and CHILAMKURTI N. Deep learning: The frontier for distributed attack detection in fog-to-things computing[J]. IEEE Communications Magazine, 2018, 56(2): 169–175. doi: 10.1109/MCOM.2018.1700332
[9]	LIU Lumin, ZHANG Jun, SONG Shenghui, et al. Client-edge-cloud hierarchical federated learning[C]. ICC 2020 - 2020 IEEE International Conference on Communications (ICC), Dublin, Ireland, 2020: 1–6.
[10]	SATTLER F, WIEDEMANN S, MüLLER K R, et al. Robust and communication-efficient federated learning from Non-i. i. d. data[J]. IEEE Transactions on Neural Networks and Learning Systems, 2020, 31(9): 3400–3413. doi: 10.1109/TNNLS.2019.2944481
[11]	SUN Jun, CHEN Tianyi, GIANNAKIS G B, et al. Lazily aggregated quantized gradient innovation for communication-efficient federated learning[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(4): 2031–2044. doi: 10.1109/TPAMI.2020.3033286
[12]	CHEN Tianyi, SUN Yuejiao, and YIN Wotao. Communication-adaptive stochastic gradient methods for distributed learning[J]. IEEE Transactions on Signal Processing, 2021, 69: 4637–4651. doi: 10.1109/TSP.2021.3099977
[13]	LU Xiaofeng, LIAO Yuying, LIO P, et al. Privacy-preserving asynchronous federated learning mechanism for edge network computing[J]. IEEE Access, 2020, 8: 48970–48981. doi: 10.1109/ACCESS.2020.2978082
[14]	MCMAHAN H B, MOORE E, RAMAGE D, et al. Communication-efficient learning of deep networks from decentralized data[C]. The 20th International Conference on Artificial Intelligence and Statistics, Fort Lauderdale, USA, 2016: 1273–1282.