This paper introduces PQR, a framework designed to generate diverse and realistic user queries that elicit failures in LLM-based QA agents, going beyond existing methods that primarily focus on adversarial users. PQR operates through iterative query and prompt refinement modules to create realistic test scenarios that expose agent vulnerabilities.
This paper introduces Reflection-Augmented Scaling (RAS) for executable Cypher query generation, leveraging prior execution feedback through in-context learning. RAS reduces the Query Execution Error Rate by 41-50%, significantly outperforming Independent Scaling.
This paper proposes an evidence-based model to generate queries from query-free summarization datasets, addressing the challenge of finding suitable datasets for Query-Focused Summarization (QFS). Experimental results indicate that summaries generated using these evidence-based queries achieve competitive ROUGE scores, supporting their effectiveness for the QFS task.