Publication
HLR-SQL: Human-like reasoning for Text-to-SQL with the human in the loop
Timo Eckmann; Matthias Urban; Jan-Micha Bodensohn; Carsten Binnig
In: Information Systems (IS), Vol. 138, Pages 1-9, Information Systems, 2026.
Abstract
Recent LLM-based approaches have achieved impressive results on Text-to-SQL benchmarks such as Spider and
Bird. However, these benchmarks do not accurately reflect the complexity typically encountered in real-world
enterprise scenarios, where queries often span multiple tables. In this paper, we introduce HLR-SQL, a new
approach designed to handle such complex enterprise SQL queries. Unlike existing methods, HLR-SQL imitates
Human-Like Reasoning with LLMs by incrementally composing queries through a sequence of intermediate
steps, gradually building up to the full query. This is an extended version of Eckmann et al. (2025). The
new contributions are centered around incorporating human feedback directly into the reasoning process of
HLR-SQL. We evaluate HLR-SQL on a newly constructed benchmark, Spider-HJ, which systematically increases
query complexity by splitting tables in the original Spider dataset to raise the average join count needed by
queries. Our experiments show that state-of-the-art models experience up to a 70% drop in execution accuracy
on Spider-HJ, while HLR-SQL achieves a 9.51% improvement over the best existing approaches on the Spider
leaderboard. Finally, we extended HLR-SQL to incorporate human feedback directly into the reasoning process
by allowing the LLM to selectively ask for human help when faced with ambiguity or execution errors. We
demonstrate that including the human in the loop in this way yields significantly higher accuracy, particularly
for complex queries.
1.