基于强化学习的自适应中间件在线更新机制研究

J4 ›› 2014, Vol. 36 ›› Issue (08): 1462-1468.

基于强化学习的自适应中间件在线更新机制研究

王建军,刘玉林

(河北经贸大学现代教育技术中心，河北石家庄 050061)

收稿日期:2012-12-10 修回日期:2013-03-07 出版日期:2014-08-25 发布日期:2014-08-25

Online updating of self-adaptive middleware
based on reinforcement learning

WANG Jianjun，LIU Yulin

(Center of Modern Education Technology,Hebei University of Economics and Business,Shijiazhuang 050061,China)

Received:2012-12-10 Revised:2013-03-07 Online:2014-08-25 Published:2014-08-25

摘要/Abstract

摘要：

自适应中间件框架一般根据预先定义的策略、按照监控、分析、决策、执行的流程实现对开放可变系统的闭环控制。但是，传统的自适应框架基于离线的闭环控制，即在提供自适应服务的同时，自身的决策模型不能随实时的环境变化而更新。针对该问题提出一种基于强化学习的自适应中间件的在线更新方案，解决自适应策略的冲突消解、系统实时效用评估问题，并设计一种基于强化学习的自适应策略在线学习更新方法，增强了自适应中间件的智能性、灵活性和应变能力。最后实现了相应的支撑系统OUSAM并在其上验证了该机制的有效性和可行性。

关键词: 自适应中间件, 在线更新, 智能决策, 强化学习

Abstract:

One common approach of selfadaptive middleware is to incorporate a control loop that monitors, analyzes, decides and executes over a target system with predefined strategies. Such approach is an offline adaptation where strategies or adaptive models are statically determined so as not to change with environment. Aiming at the problem, an online updating mechanism of selfadaptive middleware based on reinforcement learning is proposed to solve the problems of conflict resolution and realtime system effectiveness evaluation, and an online updating method of selfadaptive policy based on reinforcement learning is designed, thus enhancing intelligence, flexibility and reaction capability. Finally, the corresponding system OUSAM is implemented and the effectiveness and feasibility of the mechanism is validated on OUSAM.

Key words: self-adaptive middleware;online updating;intelligent decision;reinforcement learning

王建军,刘玉林. 基于强化学习的自适应中间件在线更新机制研究[J]. J4, 2014, 36(08): 1462-1468.

WANG Jianjun，LIU Yulin. Online updating of self-adaptive middleware
based on reinforcement learning [J]. J4, 2014, 36(08): 1462-1468.

参考文献［14］

［1］	Kephart J O, Chess D M. The vision of autonomic computing［J］. Computer, 2003,36(1):4150.
［2］	Cheng S.Rainbow:Costeffective software architecturebased selfadaptation［D］. Pittsburgh:Carnegie Mellon University, 2008.
［3］	Rahul R, Cheng S, David G, et al. Improving architecturebased selfadaptation using preemption［C］∥Proc of the Workshop on SelfOrganizing Architectures, 2010:2137.
［4］	Ahmed E,Naeem E,Sam M.FUSION:A framework for engineering selftuning selfadaptive software systems［C］∥Proc of the 18th ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2010:716.
［5］	Kim D, Park S. Reinforcement learningbased dynamic adaptation planning method for architecturebased selfmanaged software［C］∥Proc of Workshop on Software for Adaptive and SelfManaging Systems, 2009:7685.
［6］	Tesauro G, Jong N K, Das R,et al. A hybrid reinforcement learning approach to autonomic resource allocation［C］∥Proc of the International Conference on Autonomic Computing, 2006:6573.
［7］	Gao Yang, Chen Shifu, Lu Xin. Survey of reinforcement learning［J］.Chinese Journal of Automation,2004,30(1):86100.(in Chinese)
［8］	Konda V. Actorcritic algorithms［D］. Cambridge:Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, 2002.
［9］	Gu X.IDES:Selfadaptive software with online policy evolution extended from Rainbow［M］∥Computer and Information Science, 2012:181195.
［10］	Garlan D, Schmerl B, Cheng S.Software architecturebased selfadaptation［M］∥Autonomic Computing and Networking,2009:3156.
［11］	Alia M,Eide V S W,Paspallis N,et al.A utilitybased adaptivitymodel for mobile applications［C］∥Proc of the IEEE International Symposium on Ubisafe Computing, 2007:107114.
［12］	Kakousis K,Paspallis N,Papadopoulos G A.Optimizing the utility functionbased selfadaptive behavior of contextaware systems using user feedback［C］∥Proc of OTM,2008:3440.
［13］	Tesauro G, Das R, Jong N, et al. A hybrid reinforcement learning approach to autonomic resource allocation［C］∥Proc of the 3rd IEEE International Conference on Autonomic Computing,2006:6573.
［14］	Hu Haiyang,Ma Xiaoxing,Tao Xianping,et al.Research and advance of reflective middleware［J］.ChineseJournal of Computer,2005, 28(9):14071420.(in Chinese)
［15］	Benjamin B,Mathias P,Uwe B.Development and evaluation of a selfadaptive organic middleware for highly dependable systemonchips［C］∥Proc of the IARIA ’12, 2012:112117.
	附中文参考文献：
［8］	高阳,陈世福,陆鑫.强化学习研究综述［J］.自动化学报,2004,30(1):86100.
［14］	胡海洋,马晓星,陶先平,等. 反射中间件的研究与进展［J］.计算机学报,2005,28(9):14071420.

[1]	王扬, 陈智斌. 一种求解CVRP的动态图转换模型[J]. 计算机工程与科学, 2023, 45(05): 859-868.
[2]	彭坤彦, 尹翔, 刘笑竹, 李恒宇. 基于粒子群优化和深度强化学习的策略搜索方法[J]. 计算机工程与科学, 2023, 45(04): 718-725.
[3]	曾上游, 贾小硕, 李文惠. 基于双线程LSTM在线更新的视频追踪算法[J]. 计算机工程与科学, 2021, 43(03): 480-485.
[4]	童钊, 叶锋, 刘碧篮, 邓小妹, 梅晶, 刘宏. 移动边缘计算中多约束下的任务卸载和资源分配算法[J]. 计算机工程与科学, 2020, 42(10高性能专刊): 1869-1879.
[5]	蔡钺, 游进国, 丁家满. 基于近端策略优化与对抗学习的对话生成[J]. 计算机工程与科学, 2020, 42(09): 1680-1689.
[6]	官蕊, 丁家满, 贾连印, 游进国, 姜瑛, . 基于强化学习的多样性文档排序算法[J]. 计算机工程与科学, 2020, 42(09): 1697-1703.
[7]	韩虎, 孙天岳, 赵启涛. 引入自编码机制对抗网络的文本生成模型[J]. 计算机工程与科学, 2020, 42(09): 1704-1710.
[8]	周碧莹1，王爱平1，费长江2，虞万荣2，赵宝康2. 基于强化学习的卫星网络资源调度机制[J]. 计算机工程与科学, 2019, 41(12): 2134-2142.
[9]	万谦1,2，刘玮1,2，徐龙龙1,2，郭竞知1,2. 基于Q-learning的不确定环境BDI Agent最优策略规划研究[J]. 计算机工程与科学, 2019, 41(01): 166-172.
[10]	宋佳佳，王作为. 基于有效实例的改进U树算法[J]. 计算机工程与科学, 2019, 41(01): 185-190.
[11]	闫雪飞,李新明,刘东,刘德生,李强. 基于强化学习的体系对抗仿真战役层次指控算法[J]. 计算机工程与科学, 2018, 40(08): 1511-1520.
[12]	刘晓建,任怡,戴华东. 基于计数副本的传统操作系统代码空闲状态监测技术[J]. J4, 2012, 34(5): 78-83.
[13]	黄晗文1，2，郑宇3. 强化学习中基于定性模型的知识传递方法[J]. J4, 2011, 33(6): 118-124.
[14]	沈〓乐，毛新军，董孟高. 基于强化学习的自适应多Agent系统的构造[J]. J4, 2011, 33(12): 72-77.
[15]	李琼，郭御风，蒋艳凰. 基于强化学习的智能I/O调度算法[J]. J4, 2010, 32(7): 58-61.