A Reinforcement Learning-Based Multi-Agent System for Advanced Network Attack Prediction

Stephen Kahara Wanjau¹ , Stephen Njenga Thiiru²

Section:Research Paper, Product Type: Journal-Paper
Vol.12 , Issue.6 , pp.14-25, Dec-2024

Online published on Dec 31, 2024

Copyright © Stephen Kahara Wanjau, Stephen Njenga Thiiru . This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
 

View this paper at   Google Scholar | DPI Digital Library

XML View     PDF Download

How to Cite this Paper

Abstract :
This paper addresses the challenges of traditional Network Intrusion Detection Systems (NIDS) in handling the increasing complexity and volume of modern cyberattacks. The authors suggest a novel multi-agent deep reinforcement learning (MADRL) approach, employing a deep Q-network (DQN) architecture with convolutional and fully connected layers. This architecture incorporates Target networks and Experience Replay to enhance learning and adaptation. A hierarchical reinforcement learning strategy decomposes complex intrusion detection tasks into manageable subtasks, enabling efficient exploration of high-dimensional state-action spaces. The proposed model, trained and evaluated on the CICIDS2017 dataset using a 70% training set and 30% test split and 10-fold cross-validation, achieves exceptional performance. It attains 97.71% accuracy, 98.34% recall, 97.29% precision, and 96.76% F1-score after 50 iterations, surpassing existing NIDS solutions in comparative analysis. The model`s strength lies in its ability to effectively mimic environmental characteristics through multi-agent learning, leading to robust detection of intricate attack patterns. Furthermore, our approach demonstrates strong generalization capabilities on unseen data, indicating its potential for real-world deployment. This research contributes significantly to the evolution of intelligent network security systems by introducing an innovative MADRL framework. Future research directions include implementing the solution in real-time network environments, expanding the agent network, and extending the model`s application to outlier detection and software-defined networking. This work lays the foundation for future advancements in cyber threat detection and mitigation, paving the way for more robust and adaptive network security solutions.

Key-Words / Index Term :
Network Intrusion Detection Systems, Multi-Agent Systems, Deep Reinforcement Learning, Deep Q-Network, Cybersecurity, Machine Learning

References :
[1] H. Liao, C. Lin, Y. Lin and K. Tung, "Intrusion detection system: a comprehensive review," Journal of Network Computing Applications, Vol. 36, No. 1, pp. 16-24, 2013.
[2] K. Harmanpreet and K. Harjot, "Using Multi-Agent Systems for Intrusion Detection in Computer Networks: A Glance," International Journal of Advanced Research in Computer Science, Vol. 9, No. 2, pp. 497-500, 2018.
[3] I. Sharafaldin, A. Gharib, A. H. Lashkari and A. Ghorbani, "Towards a Reliable Intrusion Detection Benchmark Dataset," Journal of Software Networking, Vol. 2017, No. 1, pp. 177–200, 2017.
[4] Y. Wu, D. Wei and J. Feng, "Network Attacks Detection Methods Based on Deep Learning: A Survey," Security and Communication Networks, Vol. 2020, No. Article ID 8872923, pp. 17, 2020.
[5] F. Louati and F. Ktata, "A deep learning-based multi-agent system for intrusion detection," SN Applied Sciences, Vol. 2, No. 675, pp. 1-13, 2020.
[6] V. Jyothsna and K. Prasad, "Anomaly-Based Intrusion Detection System," in Computer and Network Security, IntechOpen, pp. 1-15, 2019.
[7] S. A. Althubiti, E. M. Jones and K. Roy, "LSTM for Anomaly-Based Network Intrusion Detection," in 2018 28th International Telecommunication Networks and Applications Conference (ITNAC), Sydney, NSW, 2018.
[8] M. Wooldridge and N. R. Jennings, "Intelligent agents: theory and practice," The Knowledge Engineering Review, Vol. 10, No. 2, pp. 115-152, 1995.
[9] J. Fox, M. Beveridge and D. Glasspool, "Understanding intelligent agents: analysis and synthesis," AI Communications, Vol. 16, No. 3, pp. 139-152, 2003.
[10] K. Monu and W. Carson, "Intelligent Agents as a Modeling Paradigm," in ICIS 2005 Proceedings, Las Vegas, NV, USA, 2005.
[11] G. Weiss, "Agent Orientation in Software Engineering," Knowledge Engineering Review, Vol. 16, No. 4, pp. 349-373, 2002.
[12] A. Hrebennyk and E. Trunova, "Modelling a Multi-agent Protection System of an Enterprise Network," ISIJ Monitor, Vol. 46, No. 3, pp. 337-340, 2020.
[13] F. Derakhshan and S. Yousefi, "A review on the applications of multiagent systems in wireless sensor networks," International Journal of Distributed Sensor Networks, Vol. 15, No. 5, pp. 1-19, 2019.
[14] Á. Herrero and E. Corchado, "Multiagent Systems for Network Intrusion Detection: A Review," Computational Intelligence in Security for Information Systems. Advances in Intelligent and Soft Computing, Springer, Berlin, Heidelberg, pp. 143-154, 2009.
[15] T. T. Nguyen, N. D. Nguyen and S. Nahavandi, "Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications," IEEE Transactions on Cybernetics, Vol. 50, No. 9, pp. 3826-3839, 2020.
[16] Y. Shoham, R. Powers and T. Grenager, "Multi-agent reinforce-ment learning: a critical survey. Tech. rep.," Stanford University, 2003.
[17] R. Lowe, Y. Wu, A. Tamar, J. Harb, O. Abbeel and I. Mordatch, "Multi-agent actor-critic for mixed cooperative-competitive environments.," in Advances in neural information processing systems, pp. 6382-639, 2017.
[18] Y. Miyashita and T. Sugawara, "Analysis of coordinated behavior structures with multi-agent deep reinforcement learning," Applied Intelligence, Vol. 51, pp. 1069-1085, 2021.
[19] N. Liu, S. Liu, R. Li and Y. Liu, "A Network Intrusion Detection Model Based on Immune Multi-Agent," International Journal of Communications, Network and System Sciences, Vol. 2, No. 6, pp. 569-574, 2009.
[20] S. Ouiazzane, F. Barramou and M. Addou, "Towards a Multi-Agent based Network Intrusion Detection System for a Fleet of Drones," International Journal of Advanced Computer Science and Applications (IJACSA), Vol. 11, No. 10, pp. 351-362, 2020.
[21] Y. Zhang and W. Lee, "Intrusion detection in wireless ad-hoc networks," in Proceedings of the 6th annual international conference on mobile computing and networking, MobiCom, Boston, MA, 2000.
[22] D. Krishnan, "A Distributed Self-Adaptive Intrusion Detection System for Mobile Ad-hoc Networks Using Tamper Evident Mobile Agents," Procedia Computer Science, Vol. 46, pp. 1203-1208, 2015.
[23] M. Riecker, S. Biedermann, R. El Bansarkhani and M. Hollick, "Lightweight energy consumption-based intrusion detection system for wireless sensor networks," International Journal of Information Security, Vol. 14, pp. 155-167, 2015.
[24] H. Alavizadeh, H. Alavizadeh and J. Jang-Jaccard, "Deep Q-Learning Based Reinforcement Learning Approach for Network Intrusion Detection," Computers, Vol. 11, No. 41, pp. 1-19, 2022.
[25] H. Benaddi, K. Ibrahimi, A. Benslimane and J. Qadir, "A Deep Reinforcement Learning Based Intrusion Detection System (DRL-IDS) for Securing Wireless Sensor Networks and Internet of Things," in Wireless Internet. WiCON 2019. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, Vol. 317, Springer, Cham., 2020.
[26] K. Sethi, Y. V. Madhav, R. Kumar and P. Bera, "Attention based multi-agent intrusion detection systems using reinforcement learning," Journal of Information Security and Applications, Vol. 61, 2021.
[27] O. Meyer, M. Hesenius and V. Gruhn, "Using Concepts to Understand Intelligent Agents," Proceedings of the AAAI 2020 Spring Symposium on Combining Machine Learning and Knowledge Engineering in Practice (AAAI-MAKE 2020), pp. 9, 2020.
[28] E. A. Feinberg and A. Shwartz, “Handbook of Markov Decision Processes: Methods and Applications,” Springer Science & Business Media, 2012.
[29] X. Li, H. Zhong and M. L. Brandeau, "Quantile Markov Decision Processes," Operations Research, Vol. 70, No. 3, 2021.
[30] A. Gudimella, R. Story, M. Shaker, R. Kong, M. Brown, V. Shnayder and M. Campos, "Deep Reinforcement Learning for Dexterous Manipulation with ConceptNetworks," arXiv preprint, p. arXiv:1709.06977, 2017.
[31] T. D. Kulkarni, K. Narasimhan, A. Saeedi and J. Tenenbaum, "Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation," in Advances in neural information processing systems, pp. 3675-3683, 2016.
[32] M. Zhou, Y. Chen, Y. Wen, Y. Yang, Y. Su, W. Zhang, D. Zhang and J. Wang, "Factorized Q-Learning for Large-Scale Multi-Agent Systems," arXiv Preprints, p. arXiv:1809.03738v4, 2019.
[33] L. Bus?oniu, R. Babu?ska and B. De Schutter, "Multi-agent reinforcement learning: An overview," in Innovations in Multi-Agent Systems and Applications, Berlin, Germany, Springer, pp. 183-221, 2010.
[34] C. Claus and C. Boutilier, "The dynamics of reinforcement learning in cooperative multiagent systems," in Proceedings of the 15th National Conference on Artificial Intelligence (AAAI) and 10th Innovative Applications of Artificial Intelligence Conference, Madison, WI, US, 1998.
[35] I. Goodfellow, Y. Bengio and A. Courville, “Deep Learning”, MIT Press, 2016.
[36] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra and M. Riedmiller, "Playing Atari with Deep Reinforcement Learning," arXiv Preprints, p. arXiv:1312.5602, 2013.
[37] M. Tokic, "Adaptive ?-greedy exploration in reinforcement learning based on value differences," in Advances in Artificial Intelligence, Berlin / Heidelberg, Springer, pp. 203-210, 2010.
[38] E. Suwannalai and C. Polprasert, "Network Intrusion Detection Systems Using Adversarial Reinforcement Learning with Deep Q-network," in 2020 18th International Conference on ICT and Knowledge Engineering (ICT&KE), Bangkok, Thailand, 2020.
[39] N. Singh, S. Krishan and U. K. Singh, "An Enhanced Multi-Agent based Network Intrusion Detection System using Shadow Log," International Journal of Computer Applications, Vol. 100, No. 9, pp. 1-5, 2014.