AutoWAFuzzer: An adaptive framework for web application firewall penetration testing with multi-agent system and RAG-enabled reinforcement learning

Web Application Firewalls (WAFs) are becoming increasingly sophisticated; however, this also presents significant limitations for current penetration testing tools in terms of adaptability, exploration capability, and contextual understanding. AutoWAFuzzer, a multi-agent framework that integrates Large Language Models (LLMs), Reinforcement Learning, and Retrieval-Augmented Generation (RAG), enables intelligent payload generation, real-time adaptation, and enhanced effectiveness in bypassing WAFs.” This is the focus of a recently published research article in Expert Systems with Applications (ESWA), conducted by a research team in which MSc. Nguyen Ngoc Thanh and MSc. Ung Van Giau, lecturers from the School of Computing and Information Technology, are among the main authors.

ESWA is a Q1-ranked journal, placed 5th globally in the fields of Artificial Intelligence and Intelligent Systems, with an Impact Factor of 7.5 and a CiteScore of 15.0. The article, titled “AutoWAFuzzer: An Adaptive Framework for Web Application Firewall Penetration Testing Based on Multi-Agent Systems and RAG-Assisted Reinforcement Learning,” demonstrates that AutoWAFuzzer significantly outperforms existing methods, opening up a new direction for automated penetration testing.

Research Summary

Web Application Firewalls (WAFs) play a crucial role in mitigating web-based threats such as SQL Injection (SQLi) and Cross-Site Scripting (XSS). However, the increasing complexity of WAF detection mechanisms has posed significant challenges for penetration testing tools. Current fuzzers based on Machine Learning and Reinforcement Learning often face three major limitations: (1) reliance on static training datasets, which makes them less flexible when encountering new WAF rules; (2) the use of single-agent architectures, which limits their ability to explore diverse attack strategies; and (3) limited contextual awareness due to the lack of integration with real-world threat intelligence. To address these challenges, we propose AutoWAFuzzer, an adaptive multi-agent framework that integrates Large Language Models (LLMs), Reinforcement Learning (RL), and Retrieval-Augmented Generation (RAG). AutoWAFuzzer decomposes the testing process into multiple modules, including an LLM-based payload generation module, a Reinforcement Learning-based policy optimizer, a reward module that simulates WAF responses, and a RAG module that continuously retrieves threat context from reputable sources such as MISP. This design enables parallel strategy exploration, context-aware payload generation, and continuous policy refinement through a closed-loop feedback mechanism among the modules. Experimental evaluations on both rule-based and Machine Learning-based WAFs — including ModSecurity, Naxsi, WAF-Brain, and CloudGuard — show that AutoWAFuzzer significantly outperforms previous methods in terms of WAF bypass rate, adaptability, and generalization capability, thereby contributing to the advancement of automated WAF penetration testing.

Read the full article at: https://doi.org/10.1016/j.eswa.2026.132546

Mr. Nguyen Ngoc Thanh and Mr. Ung Van Giau – Lecturers at School of Computing and Information Teachnology, EIU

Sharing more about the research, Mr. Nguyen Ngoc Thanh and Mr. Ung Van Giau stated:

In the context of increasingly important online services, protecting web applications has become a key issue in digital security. Web Application Firewalls (WAFs) serve as a critical layer of defense, helping detect and prevent attacks such as SQL Injection (SQLi), Cross-Site Scripting (XSS), and Remote Code Execution (RCE). However, testing WAFs has become increasingly challenging as modern WAF systems have been significantly improved.

In response to this challenge, the research team developed AutoWAFuzzer, an adaptive and automated WAF penetration testing framework that integrates Large Language Models (LLMs), Reinforcement Learning, and Retrieval-Augmented Generation. Experimental results across various types of WAFs show that AutoWAFuzzer has the potential to enhance the effectiveness of automated cybersecurity testing through its ability to generate diverse payloads, learn from feedback, and leverage real-world cybersecurity knowledge.

This work was completed during our time at Eastern International University (EIU) and reflects our aspiration to further develop research directions that combine AI, Cybersecurity, Data Science, and highly applicable intelligent systems at EIU.