An Evolutionary Approach to Reinforcement Learning of Neural Network Controllers for Pendulum Tasks Using Genetic Algorithms

Hidehiko Okada¹

1. Faculty of Information Science and Engineering, Kyoto Sangyo University, Kyoto, Japan.

Section:Research Paper, Product Type: Journal-Paper
Vol.11 , Issue.1 , pp.40-46, Feb-2023

Online published on Feb 28, 2023

Copyright © Hidehiko Okada . This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
 

View this paper at   Google Scholar | DPI Digital Library

XML View     PDF Download

How to Cite this Paper

139 Views    155 Downloads    37 Downloads

Abstract :
Reinforcement learning of neural networks requires gradient-free algorithms because of the absence of labeled training data. Evolutionary algorithms, which do not rely on gradients, are a viable option for this purpose. To successfully train neural networks by evolutionary algorithms, we need to carefully choose appropriate algorithms because many algorithm variations are available. This paper experimentally evaluates the effectiveness of the Genetic Algorithm, a type of evolutionary algorithm, in training neural networks for reinforcement learning. The task selected for this evaluation is pendulum control. The results show that GA is capable of efficiently training a multilayer perceptron to maintain the pendulum in an upright position. The number of hidden units (8, 16 and 32) in the MLP was found to have no significant effect on the performance. Thus, GA could train the MLP to perform the task appropriately even with a small number of hidden units. Furthermore, the results of this study were compared with those obtained from Evolution Strategy, and it was observed that GA performed better than ES when the number of generations was given priority, while ES performed better when the population size was given priority.

Key-Words / Index Term :
Evolutionary algorithm, Genetic algorithm, Neural network, Neuroevolution, Reinforcement learning

References :
[1] T. Bäck, H.P. Schwefel, “An Overview of Evolutionary Algorithms for Parameter Optimization,” Evolutionary Computation, Vol.1, No.1, pp.1-23, 1993.
[2] D.B. Fogel, “An Introduction to Simulated Evolutionary Optimization,” IEEE Transactions on Neural Networks, Vol.5, No.1, pp.3-14, 1994.
[3] T. Bäck, “Evolutionary Algorithms in Theory and Practice: Evolution Strategies, Evolutionary Programming, Genetic Algorithms,” Oxford University Press, 1996.
[4] A.E. Eiben, R. Hinterding, Z. Michalewicz, “Parameter Control in Evolutionary Algorithms,” IEEE Transactions on Evolutionary Computation, Vol.3, No.2, pp.124-141, 1999.
[5] A.E. Eiben, J.E. Smith, “Introduction to Evolutionary Computing (2nd ed.),” Springer, 2015.
[6] C.J.C.H. Watkins, “Learning from Delayed Rewards,” PhD Thesis, Cambridge University, 1989.
[7] C.J.C.H. Watkins, P. Dayan, “Q-Learning,” Machine Learning, Vol.8, No.3, pp.279-292, 1992.
[8] R.S. Sutton, A.G. Barto, “Reinforcement Learning: An Introduction (2nd ed.),” MIT Press, 2018.
[9] H.P. Schwefel, “Evolution Strategies: A Family of Non-Linear Optimization Techniques based on Imitating Some Principles of Organic Evolution,” Annals of Operations Research, Vol.1, pp.165-167, 1984.
[10] H.G. Beyer, H.P. Schwefel, “Evolution Strategies: A Compre-hensive Introduction,” Journal Natural Computing, Vol.1, No.1, pp.3-52, 2002.
[11] D.E. Goldberg, J.H. Holland, “Genetic Algorithms and Machine Learning,” Machine Learning, Vol.3, No.2, pp.95-99, 1988.
[12] J.H. Holland, “Genetic Algorithms,” Scientific American, Vol.267, No.1, pp.66-73, 1992.
[13] M. Mitchell, “An Introduction to Genetic Algorithms,” MIT Press, 1998.
[14] K. Sastry, D. Goldberg, G. Kendall, “Genetic Algorithms,” Search Methodologies, Springer, pp.97-125, 2005.
[15] R. Storn, K. Price, “Differential Evolution – A Simple and Efficient Heuristic for Global Optimization over Continuous Spaces,” Journal of Global Optimization, Vol.11, pp.341-359, 1997.
[16] K. Price, R.M. Storn, J.A. Lampinen, “Differential Evolution: A Practical Approach to Global Optimization,” Springer Science & Business Media, 2006.
[17] S. Das, P.N. Suganthan, “Differential Evolution: A Survey of the State-of-the-art,” IEEE transactions on evolutionary computation, Vol.15, No.1, pp.4-31, 2010.
[18] H. Okada, “Evolutionary Reinforcement Learning of Neural Network Controller for Pendulum Task by Evolution Strategy,” International Journal of Scientific Research in Computer Science and Engineering, Vol.10, Issue 3, pp.13-18, 2022.
[19] D.E. Rumelhart, G.E. Hinton, R.J. Williams. “Learning Internal Representations by Error Propagation,” in D.E. Rumelhart, J.L. McClelland, and the PDP research group (editors), “Parallel Distributed Processing: Explorations in the Microstructure of Cognition,” Vol.1: Foundation. MIT Press, 1986.
[20] R. Collobert, S. Bengio, “Links Between Perceptrons, MLPs and SVMs,” Proc. of the Twenty-First International Conference on Machine Learning (ICML’04), ACM, 2004.
[21] X. Yao, Y. Liu, “A New Evolutionary System for Evolving Arti?cial Neural Networks,” IEEE Transactions on Neural Networks, Vol.8, No.3, pp.694-713, 1997.
[22] N.T. Siebel, G. Sommer, “Evolutionary Reinforcement Learning of Artificial Neural Networks,” Internatinal Journal of Hybrid Intelligent Systems, Vol.4, No.3, pp.171-183. 2007.
[23] K. Chellapilla, D.B. Fogel, “Evolving Neural Networks to Play Checkers Without Relying on Expert Knowledge,” IEEE Transactions on Neural Networks, Vol.10, No.6, pp.1382-1391, 1999.
[24] L. Cardamone, D. Loiacono and P. L. Lanzi, “Evolving Competitive Car Controllers for Racing Games with Neuro-evolution,” Proc. of 11th Annual Conference on Genetic and Evolutinary Computation, pp.1179-1186, 2009.
[25] S. Risi, J. Togelius, “Neuroevolution in Games: State of the Art and Open Challenges”, IEEE Transactions on Computational Intelligence and AI in Games, Vol.9, No.1, pp.25-41, 2017.
[26] J. Togelius, S.M. Lucas, “Evolving Controllers for Simulated Car Racing,” Proc. of 2005 IEEE Congress on Evolutionary Computation, Vol.2, pp.1906-1913, 2005.
[27] L.J. Eshelman, J.D. Schaffer, “Real-coded Genetic Algorithms and Interval-Schemata,” Foundations of Genetic Algorithms, Vol.2, pp.187-202, 1993.