Multi-Level Safety Filtering Method for LLM Generation Security

Abstract

Large Language Models (LLMs) exhibit limitations due to their static para-metric knowledge, leading to potential inaccuracies and an inability to incor-porate updated information. Retrieval-Augmented Generation (RAG) mitigates this by integrating external knowledge bases. However, existing RAG paradigms face challenges such as information forgetting in long conversational contexts and vulnerability to knowledge poisoning attacks, which compromise the accu-racy and security of generated content. This research aims to enhance the capa-bilities of LLMs augmented by external knowledge by addressing these two crit-ical issues. To tackle information forgetting during conversational query rewriting, we propose KwRewriter, a keyword-integrated query rewriting framework. It em-ploys a document keyword extractor to enrich document representations and a query rewriter that fuses generated keywords with the rewritten query, empha-sizing core user intent. For security concerns, we introduce MLSF, a multi-level safety filtering method. MLSF implements a three-stage defense: vector-level coarse screening via K-means clustering, fine screening combining keyword consensus and BERTScore semantic matching, and final fact verification using the LLM's internal knowledge. Experimental results on open-domain QA datasets demonstrate KwRewrit-er's effectiveness, significantly improving retrieval metrics like MRR and Re-call@k over strong baselines, proving its ability to alleviate information loss. MLSF was validated on datasets subjected to knowledge poisoning attacks, where it substantially increased answer accuracy and reduced the attack success rate, especially under high poisoning ratios, confirming its robust defensive capability. In conclusion, this work successfully enhances RAG systems by improving both the accuracy of retrieval through context-aware query rewriting and the security of generation via hierarchical document filtering. The proposed meth-ods were integrated into a practical medical retrieval QA system, underscoring their applicability and value in providing reliable, knowledge-grounded re-sponses. Future work will focus on optimizing keyword integration, improving method generalizability, and exploring defenses against a broader spectrum of RAG security threats.