AI-powered natural language to SQL generation for lange-scale data analysis solutions

Abstract

In enterprise-grade, high-scale data analysis scenarios involves Natural-Language-to-SQL transfor-mation is essential. Due to the vastness of table structures, general problems from large model occur during training, causing Large language model (LLM) to be unable to handle contextual information adequately, thus reducing query precision significantly.

This thesis proposes an integrated model of Artificial Intelligence (AI) and Natural language pro-cessing (NLP) to generate corresponding SQL query codes automatically based on natural language descriptions of massive datasets and large-scale data. By using Large language model orchestration, retrieval-augmented generation (RAG) and AI-workflow technology, users can conduct data analysis without technical ability.

The methodology of the intelligent text-to-SQL agent system can integrate retrieval-augmented gen-eration (RAG) and a closed-loop self-correction mechanism seamlessly. Firstly, the lightweight vec-tor-based database (Chroma DB) is used to retrieve the semantic schema, and only the top parameter most similar tables Data definition language (DDLs) are selected as the result. Next, the condensed schemata will be dynamically assembled into a structured prompt. Subsequently, these prompts help generate an initially constructed SQL statement. Finally, at the end of executing all SQL commands in one go, if there are any errors, it will automatically send a physical error log via Large language mod-el (LLM) iteration-based re-creation until a stable, ready-to-run query can be generated.

The results of thesis were evaluated through following testing. Zero-shot scenario tests in spider-cross-database benchmark (1,034 complex queries spanning different databases), compared against an LLM-alone control set for evaluating whether this boosted execution accuracy, lowered tokens about 79 per cent and greatly improved generalised performance through a specific way here described. These tested bridge semantic understanding and automatic processing to create an actual, SQL-independent analytical system for large amounts of data in enterprises, and provided not only theories but also practical applications.