Q-SEM: Fine-Grained Question-Grounded Semantic Evaluation for Text-to-SQL
Abstract
Text-to-SQL evaluation remains largely reference-centered. Execution Accuracy (EX), the dominant metric, evaluates a predicted SQL query by comparing its execution result against that of a single reference SQL. However, EX frequently fails when a prediction follows the same intended logic as the reference but differs in underspecified output behavior, when the question or schema admits multiple plausible interpretations, or when the reference SQL itself is noisy or incorrect. We therefore propose Q-SEM, a fine-grained, Question-grounded Semantic Evaluation Metric that evaluates whether a predicted SQL query preserves the intent of the natural-language question. Q-SEM combines two gold-SQL-agnostic views that assess whether the prediction covers and correctly composes the question’s semantic requirements with two gold-SQL-contrastive views that determine whether semantic differences constitute actual semantic errors with respect to the question intent. A final adjudicator resolves disagreements among the four views and produces the final acceptability decision. On the expert-labeled ROSE-VEC-BIRD benchmark, Q-SEM achieves 85.61% Cohen’s κ, obtaining the strongest agreement with human judgments and consistently outperforming deterministic metrics (EX reaches only 43.56%) and LLM-based methods across every evaluated LLM.
