Designing an AI-powered ETL Platform : a case study on 3GPP Change Request Documents

Abstract

LLMs, or large language models, can process natural language and generate content according to a prompt. The generation can be significantly improved and customized if the model is given authoritative data outside its pre-training data. This optimization and customization process is called retrieval-augmented generation, or RAG. With RAG technology, the LLM can serve as a junior consultant in multiple fields. Implementing RAG for a specific field requires RAG-Ready data, which means the data is high-quality, clean, cost-effective, and easily convertible to vector data. The challenging part is the raw data suitable for RAG, primarily free-length text in the unstructured or semi-structured form, which is difficult to extract using traditional methods. This project will design and develop an ETL platform to process raw data to RAG-Ready data. This project used Python as the programming language and Flask as the framework. The databases included GCP and a local SQL database. It utilized GitHub for version control. What distinguishes it from traditional ETL is that it uses LLM technology and implements creative methods to make LLM technology an essential part of the ETL process and achieve the expected result.