AI-powered cyber resilience: a reinforcement learning approach for automated threat hunting in 5G networks

Abstract

Abstract The rapid evolution of 5G networks has introduced unparalleled connectivity, speed, and efficiency. However, these advancements bring critical cybersecurity challenges due to increased attack surfaces, network slicing vulnerabilities, and adaptive threat vectors. Traditional IDS and ML-based mechanisms lack the adaptability required for real-time threat mitigation in such complex environments. To address this, we propose SecureNet-RL, an AI-powered cyber resilience model that leverages reinforcement learning (RL) for proactive, real-time threat hunting in 5G networks. The framework integrates Deep Q-Networks and Proximal Policy Optimization with federated multi-agent learning to enable distributed, adaptive defense across network slices. SecureNet-RL agents autonomously learn from network traffic patterns, identify anomalies, and mitigate threats without human intervention. The system incorporates adversarial robustness through reward shaping and GAN-generated attack simulations to counter evolving threats. Evaluated using DDoS, insider, and 0-day threat scenarios over NS3-based 5G slices, the model consistently achieves 95.8% detection accuracy, < 50 ms mitigation latency, and a false-positive rate of just 4.3%—surpassing traditional IDS (78.2%) and supervised ML (88.5%). It also demonstrates efficient resource utilization with 45% CPU and 1.5 GB memory, ensuring scalability in large-scale 5G deployments. Key contributions include: (1) real-time adaptive RL-based threat hunting, (2) federated learning integration for decentralized scalability, (3) adversarial robustness via reward and attack modeling, and (4) comprehensive validation on multi-modal threat datasets. These findings confirm that SecureNet-RL offers a scalable, intelligent, and self-improving cyber defense solution tailored for next-generation 5G networks.

Keywords

• Reinforcement learning
• Cyber resilience
• Automated threat hunting
• 5G security
• Anomaly detection