This research addresses RAG failures caused by insufficient or conflicting retrieved documents by treating the problem as a three-way classification task. The authors utilize hidden activations and attention-derived features to predict whether input information is sufficient, insufficient, or conflicting. They introduce a controlled benchmark dataset featuring fictitious information to evaluate these internal model signals effectively.
- Use internal model signals like hidden activations to detect RAG data quality issues.
- Classify retrieval outcomes into sufficient, insufficient, or conflicting categories.
- Leverage attention-derived features for more robust reliability detection.
- Adopt the new benchmark with fictitious data for rigorous RAG evaluation.
