基于增強學習算法的插電式燃料電池電動汽車能量管理控制策略

林歆悠; 夏玉田; 魏申申

doi:10.13374/j.issn2095-9389.2018.10.15.001

基于增強學習算法的插電式燃料電池電動汽車能量管理控制策略

doi: 10.13374/j.issn2095-9389.2018.10.15.001

林歆悠^{1), 2), ,},
夏玉田¹⁾,
魏申申¹⁾

1).
福州大學機械工程及自動化學院, 福州 350002
2).
流體動力與電液智能控制福建省高校重點實驗室(福州大學), 福州 350002

基金項目:

國家自然科學基金資助項目 51505086

詳細信息

通訊作者:
林歆悠, E-mail: linxinyoou@fzu.edu.cn

中圖分類號: TG142.71
計量
- 文章訪問數: 1104
- HTML全文瀏覽量: 405
- PDF下載量: 39
- 被引次數: 0
出版歷程
- 收稿日期: 2018-10-15
- 刊出日期: 2019-10-01

Energy management control strategy for plug-in fuel cell electric vehicle based on reinforcement learning algorithm

LIN Xin-you^{1), 2)
, ,},
XIA Yu-tian¹⁾,
WEI Shen-shen¹⁾

1).
College of Mechanical Engineering and Automation, Fuzhou University, Fuzhou 350002, China
2).
Key Laboratory of Fluid Power and Intelligent Electro-Hydraulic Control, Fuzhou University, Fuzhou 350002, China

More Information

Corresponding author: LIN Xin-you, E-mail: linxinyoou@fzu.edu.cn

摘要

摘要: 以一款插電式燃料電池電動汽車(plug-in fuel cell electric vehicle，PFCEV)為研究對象，為改善燃料電池氫氣消耗和電池電量消耗之間的均衡，實現插電式燃料電池電動汽車的燃料電池與動力電池之間的最優能量分配，考慮燃料電池汽車實時能量分配的即時回報及未來累積折扣回報，以整車作為環境，整車控制作為智能體，提出了一種基于增強學習算法的插電式燃料電池電動汽車能量管理控制策略.通過Matlab/Simulink建立整車仿真模型對所提出的策略進行仿真驗證，相比于基于規則的策略，在不同行駛里程下，電池均可保持一定的電量，整車的綜合能耗得到明顯降低，在100、200和300 km行駛里程下整車百公里能耗分別降低8.84%、29.5%和38.6%；基于快速原型開發平臺進行硬件在環試驗驗證，城市行駛工況工況下整車綜合能耗降低20.8%，硬件在環試驗結果與仿真結果基本一致，表明了所制定能量管理策略的有效性和可行性.
- 燃料電池汽車 /
- 增強學習 /
- 能量管理 /
- Q_learning算法 /
- 控制策略
Abstract: To cope with the increasingly stringent emission regulations, major automobile manufacturers have been focusing on the development of new energy vehicles. Fuel-cell vehicles with advantages of zero emission, high efficiency, diversification of fuel sources, and renewable energy have been the focus of international automotive giants and Chinese automotive enterprises. Establishing a reasonable energy management strategy, effectively controlling the vehicle working mode, and reasonably using battery energy for hybrid fuel-cell vehicles are core technologies in domestic and foreign automobile enterprises and research institutes. To improve the equilibrium between fuel-cell hydrogen consumption and battery consumption and realize the optimal energy distribution between fuel-cell systems and batteries for plug-in fuel-cell electric vehicles (PFCEVs), considering vehicles as the environment and vehicle control as an agent, an energy management strategy for the PFCEV based on reinforcement learning algorithm was proposed in this paper. This strategy considered the immediate return and future cumulative discounted returns of a fuel-cell vehicle's real-time energy allocation. The vehicle simulation model was built by Matlab/Simulink to carry out the simulation test for the proposed strategy. Compared with the rule-based strategy, the battery can store a certain amount of electricity, and the integrated energy consumption of the vehicle was notably reduced under different mileages. The energy consumption in 100 km was reduced by 8.84%, 29.5%, and 38.6% under 100, 200, and 300 km mileages, respectively. The hardware-in-loop-test was performed on the D2P development platform, and the final energy consumption of the vehicle was reduced by 20.8% under urban dynamometer driving schedule driving cycle. The hardware-in loop-test results are consistent with the simulation findings, indicating the effectiveness and feasibility of the proposed energy management strategy.
- fuel-cell vehicle /
- reinforcement learning /
- energy management /
- Q_learning algorithm /
- control strategy

HTML全文

圖 1 燃料電池汽車動力系統結構

Figure 1. Structure of the fuel cell vehicle driving system

下載: 全尺寸圖片幻燈片

圖 2 智能體和環境之間的交互過程

Figure 2. Iterative interaction between the agent and environment

下載: 全尺寸圖片幻燈片

圖 3 狀態轉移概率的計算過程

Figure 3. Calculation process of the state transfer probability

下載: 全尺寸圖片幻燈片

圖 4 基于增強學習的控制策略求解過程

Figure 4. Process of solving the control strategy based on RL

下載: 全尺寸圖片幻燈片

圖 5 Q_learning學習迭代中的百步均方差

Figure 5. 100-step mean square error in Q_learning iteration

下載: 全尺寸圖片幻燈片

圖 6 增強學習價值函數優化結果. (a)價值函數的最優值; (b)迭代后的Q值

Figure 6. Optimization results of the RL cost function: (a)optimal solution of cost function; (b) Q values after iteration

下載: 全尺寸圖片幻燈片

圖 7 燃料電池混合動力汽車仿真模型

Figure 7. Simulation model of the fuel cell hybrid electric vehicle

下載: 全尺寸圖片幻燈片

圖 8 實際車速與目標車速對比

Figure 8. Comparison between actual and target speeds

下載: 全尺寸圖片幻燈片

圖 9 不同策略下的驗證結果對比. (a)電池荷電狀態變化對比; (b)電池能耗對比; (c)氫氣消耗量對比

Figure 9. Comparison of the results for three strategies: (a)comparison of the battery SOC; (b)comparison of battery energy consumption; (c)comparison of fuel cell hydrogen consumption

下載: 全尺寸圖片幻燈片

圖 10 燃料電池系統輸出功率對比

Figure 10. Comparison of output power of fuel cell system

下載: 全尺寸圖片幻燈片

圖 11 燃料電池系統效率變化

Figure 11. Change of fuel cell system efficiency

下載: 全尺寸圖片幻燈片

圖 12 整車百公里綜合能耗對比

Figure 12. Comparison of comprehensive energy consumption for one hundred kilometers

下載: 全尺寸圖片幻燈片

圖 13 硬件在環試驗臺架

Figure 13. Test bench of the hardware in loop

下載: 全尺寸圖片幻燈片

圖 14 硬件在環試驗測試系統

Figure 14. Hardware in loop test system

下載: 全尺寸圖片幻燈片

圖 15 電機功率試驗與仿真結果對比

Figure 15. Comparison of motor power between test and simulation

下載: 全尺寸圖片幻燈片

圖 16 電池功率試驗與仿真結果對比

Figure 16. Comparison of battery power between test and simulation

下載: 全尺寸圖片幻燈片

圖 17 燃料電池功率試驗與仿真結果對比

Figure 17. Comparison of fuel cell power between test and simulation

下載: 全尺寸圖片幻燈片

圖 18 電池荷電狀態和整車綜合能耗仿真與試驗結果對比

Figure 18. Comparison of battery SOC and vehicle integrated energy consumption between test and simulation

下載: 全尺寸圖片幻燈片

表 1 整車基本參數

Table 1. Basic parameters for vehicle

整備質量/kg	軸距/mm	滾動半徑/mm	空氣阻力系數	迎風面積/m²	傳動系效率	主減速比	驅動電機最大功率/kW	燃料電池系統最大功率/kW	動力電池容量/(A·h)
1400	1700	301	0.284	1.97	0.95	4.226	75	65	40

下載: 導出CSV

表 2 Q-learning算法在Matlab中的計算流程

Table 2. Computing process of Q-learning algorithms in Matlab

??初始化Q(s, a)，s∈S，a∈A(s), 任意Q(s, a)=0
初始化狀態S(P_m(t), SOC(t), v(t))
重復(對每一次迭代中的每一步)：
??根據狀態S選取一個動作A(P_b(t))執行
??執行完A動作后觀察回報值R和新的狀態S′
?? $ Q(s, a) \leftarrow Q(s, a)+\eta\left(r+\gamma \max\limits _{a^{\prime}} Q\left(s^{\prime}, a^{\prime}\right)-Q(s, a)\right)$
?? S←S′
循環直到S終止

下載: 導出CSV

表 3 不同行駛里程的仿真運行結果對比

Table 3. Comparison of simulation results under different mileages

控制策略	電池荷電狀態	氫氣消耗/kg	綜合能耗/(kW·h)	能耗降低/%
規則, 50 km	0.5491	0	7.56	[-]
等效最小, 50 km	0.5491	0	7.56	0
增強學習, 50 km	0.5521	0.006	7.56	0
規則, 100 km	0.3022	0.106	16.07	[-]
等效最小, 100 km	0.2225	0.014	14.65	8.84
增強學習, 100 km	0.3472	0.114	14.76	8.15
規則, 150 km	0.2072	0.382	27.19	[-]
等效最小, 150 km	0.2104	0.297	22.28	18.1
增強學習, 150 km	0.2076	0.306	22.53	17.1
規則, 200 km	0.2856	0.833	39.7	[-]
等效最小, 200 km	0.2014	0.577	28.75	27.6
增強學習, 200 km	0.2012	0.503	27.99	29.5
規則, 300 km	0.2401	1.538	62.05	[-]
等效最小, 300 km	0.2021	1.158	41.85	32.3
增強學習, 300 km	0.2013	0.913	38.35	38.2

下載: 導出CSV

表 4 整車的綜合百公里能耗

Table 4. Comprehensive energy consumption for one hundred kilometers ?kW·h

里程/km	規則	等效最小	增強學習
50	15.12	15.12	15.12
100	16.07	14.65	14.76
150	18.13	14.85	15.02
200	19.85	14.38	14
300	20.83	13.95	12.78

下載: 導出CSV

www.77susu.com

參考文獻(19)

[1]	Xu L F, Hua J F, Bao L, et al. Optimized strategy on equivalent hydrogen consumption for fuel cell hybrid electric bus. China J Highway Transport, 2009, 22(1): 104 doi: 10.3321/j.issn:1001-7372.2009.01.017 徐梁飛, 華劍鋒, 包磊, 等. 燃料電池混合動力客車等效氫耗優化策略. 中國公路學報, 2009, 22(1): 104 doi: 10.3321/j.issn:1001-7372.2009.01.017
[2]	Yun H T, Liu S D, Zhao Y L, et al. Energy management for fuel cell hybrid vehicles based on a stiffness coefficient model. Int J Hydrogen Energy, 2015, 40(1): 633 doi: 10.1016/j.ijhydene.2014.10.135
[3]	Oldenbroek V, Verhoef L A, Van Wijk A J M. Fuel cell electric vehicle as a power plant: Fully renewable integrated transport and energy system design and analysis for smart city areas. Int J Hydrogen Energy, 2017, 42(12): 8166 doi: 10.1016/j.ijhydene.2017.01.155
[4]	Yang W B, Chen Q S, Tian G Y, et al. Shifting schedule and torque distribution strategy for the plug-in hybrid electric vehicle. J Mech Eng, 2013, 49(14): 91 https://www.cnki.com.cn/Article/CJFDTOTAL-JXXB201314015.htm 楊偉斌, 陳全世, 田光宇, 等. 插電式混合動力汽車換擋規律及轉矩分配策略. 機械工程學報, 2013, 49(14): 91 https://www.cnki.com.cn/Article/CJFDTOTAL-JXXB201314015.htm
[5]	Ahmadi L, Croiset E, Elkamel A, et al. Effect of socio-economic factors on EV/HEV/PHEV adoption rate in Ontario. Technol Forecast Social Change, 2015, 98: 93 doi: 10.1016/j.techfore.2015.06.012
[6]	Montazeri-Gh M, Mahmoodi-K M. An optimal energy management development for various configuration of plug-in and hybrid electric vehicle. J Cent South Univ, 2015, 22(5): 1737 doi: 10.1007/s11771-015-2692-6
[7]	Qin D T, Zhao X Q, Su L, et al. Variable parameter energy management strategy for plug-in hybrid electric vehicle. China J Highway Transport, 2015, 28(2): 112 doi: 10.3969/j.issn.1001-7372.2015.02.014 秦大同, 趙新慶, 蘇嶺, 等. 插電式混合動力汽車變參數能量管理策略. 中國公路學報, 2015, 28(2): 112 doi: 10.3969/j.issn.1001-7372.2015.02.014
[8]	Lin C C, Peng H, Jeon S, et al. Control of a hybrid electric truck based on driving pattern recognition//Proceedings of the 2002 Advanced Vehicle Control Conference. Hiroshima, 2002: 9
[9]	Lin X Y, Sun D Y. Development of control strategy for a series-parallel hybrid electric city bus based on roadway type recognition. China Mech Eng, 2012, 23(7): 869 doi: 10.3969/j.issn.1004-132X.2012.07.023 林歆悠, 孫冬野. 基于工況識別的混聯式混合動力客車控制策略研究. 中國機械工程, 2012, 23(7): 869 doi: 10.3969/j.issn.1004-132X.2012.07.023
[10]	Zhu Y, Wu Z H, Tian G Y, et al. An energy management strategy for fuel cell hybrid electric vehicle based on Markov decision process. Autom Eng, 2006, 28(9): 798 doi: 10.3321/j.issn:1000-680X.2006.09.003 朱元, 吳志紅, 田光宇, 等. 基于馬爾可夫決策理論的燃料電池混合動力汽車能量管理策略. 汽車工程, 2006, 28(9): 798 doi: 10.3321/j.issn:1000-680X.2006.09.003
[11]	Shi Y Q, He B, Cao G J, et al. A study on the energy management strategy for fuel cell electric vehicle based on instantaneous optimization. Autom Eng, 2008, 30(1): 30 doi: 10.3321/j.issn:1000-680X.2008.01.007 石英喬, 何彬, 曹桂軍, 等. 燃料電池混合動力瞬時優化能量管理策略研究. 汽車工程, 2008, 30(1): 30 doi: 10.3321/j.issn:1000-680X.2008.01.007
[12]	Wang Q P, Du S Y, Li L, et al. Real-time energy management strategy for plug-in hybrid electric bus on particle swarm optimization algorithm. J Mech Eng, 2017, 53(4): 77 https://www.cnki.com.cn/Article/CJFDTOTAL-JXXB201704012.htm 王欽普, 杜思宇, 李亮, 等. 基于粒子群算法的插電式混合動力客車實時策略. 機械工程學報, 2017, 53(4): 77 https://www.cnki.com.cn/Article/CJFDTOTAL-JXXB201704012.htm
[13]	Lin X Y, Feng Q G, Zhang S B. Global optimal discrete equivalent factor of equivalent fuel consumption minimization strategy based energy management strategy for a series-parallel plug-in hybrid electric vehicle. J Mech Eng, 2016, 52(20): 102 https://www.cnki.com.cn/Article/CJFDTOTAL-JXXB201620014.htm 林歆悠, 馮其高, 張少博. 等效因子離散全局優化的等效燃油瞬時消耗最小策略能量管理策略. 機械工程學報, 2016, 52(20): 102 https://www.cnki.com.cn/Article/CJFDTOTAL-JXXB201620014.htm
[14]	Hu Z Y, Li J Q, Xu L F, et al. Multi-objective energy management optimization and parameter sizing for proton exchange membrane hybrid fuel cell vehicles. Energy Convers Manage, 2016, 129: 108 doi: 10.1016/j.enconman.2016.09.082
[15]	Xu L F, Mueller C D, Li J Q, et al. Multi-objective component sizing based on optimal energy management strategy of fuel cell electric vehicles. Appl Energy, 2015, 157: 664 doi: 10.1016/j.apenergy.2015.02.017
[16]	Han J H, Park Y, Kum D. Optimal adaptation of equivalent factor of equivalent consumption minimization strategy for fuel cell hybrid electric vehicles under active state inequality constraints. J Power Sources, 2014, 267: 491 doi: 10.1016/j.jpowsour.2014.05.067
[17]	Murgovski N, Johannesson L M, Sj?berg J. Engine on/off control for dimensioning hybrid electric powertrains via convex optimization. IEEE Trans Veh Technol, 2013, 62(7): 2949 doi: 10.1109/TVT.2013.2251920
[18]	Zheng C H, Oh C E, Park Y I, et al. Fuel economy evaluation of fuel cell hybrid vehicles based on equivalent fuel consumption. Int J Hydrogen Energy, 2012, 37(2): 1790 doi: 10.1016/j.ijhydene.2011.09.147
[19]	Huang Y J, Wang H, Khajepour A, et al. Model predictive control power management strategies for HEVs: a review. J Power Sources, 2017, 341: 91 doi: 10.1016/j.jpowsour.2016.11.106