基于改进PPO算法的混合动力汽车能量管理策略
2025-08-09马超孙统曹磊杨坤胡文静
中图分类号:U469.72 文献标识码:A DOI:10.7535/hbkd.2025yx03001
Energy management strategy for hybrid electric vehicle based on improved PPO algorithm
MA Chao, SUN Tong,CAO Lei,YANG Kun, HU Wenjing (Collgeof TransportationandVehicle Engineering,Shandong Universityof Technology,Zibo,Shandong 2550oo,China
Abstract:Inorder toimprovetheeconomyof power-split hybrid electric vehicle(HEV),alongitudinal dynamics modelof the entire HEVvehicle was established,and an energy management strategy(EMS)basedonstrategy entropy optimization withanimproved proximalpolicyoptimization(PPO)algorithmwasproposed.Thealgorithmicframework wassimplified by employing an experiencepoling mechanism based on traditional PPO algorithm,andonlyone deep neural network was used forinteractivetrainingandupdating toreducethecomplexityofparametersynchronizationinthepolicynetwork.Inorderto efectively explore theenvironmentand learnmoreeficientstrategies,thestrategyentropywasadded tothelossfunctionto promotetheintellgencetostrikeabalancebetweenexplorationandutilizationandtoavoid prematureconvergenceofstrategies tolocaloptimal solutions.TheresultsshowthattheEMS basedontheimprovedPPOalgorithmwith single-policynetwork maintains thestateof charge(SOC)of the battry more efectivelythantheEMS basedonthedual-strategy network PPO under both UDDS and NEDC driving cycle. Additionally,the equivalent fuel consumption is reduced by 8.5% and 1.4% , respectively,achieving energy-saving efectscomparable to the EMS basedon the dynamic programming (DP)algorithm. The proposed improvedPPOalgorithmcaneffectivelyenhance the fueleconomyof hybridvehiclesand provideareference for the design and development of EMS for hybrid vehicles.
Keywords:vehicleenginering;hybrid electric vehicle;energy managementstrategy;deepreinforcementlearning;proximal policy optimization
汽车保有量的迅速增长,不仅给国家能源安全带来巨大的挑战,环境污染问题也日益凸显[1],而混合动力汽车(hybrid electric vehicle,HEV)被认为是减排的重要角色。HEV能够通过适当的能量管理策略(energy management strategy,EMS),使得多个动力源的合作高效运行,从而降低燃料消耗和温室气体排放[2]。HEV的EMS可以分为基于规则的策略、基于优化的策略和基于学习的策略[3]。基于规则的策略是工程领域实现实时控制的最常用方法,但是高效策略的设计依赖于工程师的专业水平[4]。基于优化的策略是将能量管理构建成一个最优化问题,求解可行域中的最优解或次优解,在一定程度上提高EMS的燃油经济性,但是在复杂的情况下难以实时求解并在线应用[5-6]。
近年来,随着人工智能的快速发展,基于学习的方法开始应用在 HEV 的能量管理问题中[7]。LIN 等[8]首次将强化学习Q-learning算法应用到HEV的能量管理问题中,EMS问题被建模为马尔可夫决策过程,智能体与环境进行交互学习并更新策略,不仅减少了计算时间,还提高了车辆的燃油经济性。但是由于离散状态的限制,Q-learning 算法很难处理高维度的问题。WU等[9]提出了一种用于EMS 的深度Q学习(deep Q-network,DQN)算法,使用深度神经网络近似动作值函数,替代Q-learning算法中的表格存储,解决了维数诅咒问题。WANG等[10]采用改进的参数化DQN算法,降低了 3.1% 的驱动成本,并且有效延长了电池寿命。WU等[利用深度确定性策略梯度(deep deterministic policy gradient,DDPG)开发了插电式混合动力公交车的 EMS,能够连续输出控制动作而不需要将动作空间离散化。……
