AI-powered Legal Virtual Assistant: Utilizing RAG-optimized LLM for Housing Dispute Resolution in Finland.
Rafat, Md Irfan (2024)
2024
All rights reserved. This publication is copyrighted. You may download, display and print it for Your own personal use. Commercial use is prohibited.
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi:amk-2024052816910
https://urn.fi/URN:NBN:fi:amk-2024052816910
Tiivistelmä
Driven by advancements in Artificial Intelligence (AI), a wave of transformation is sweeping across numerous industries, and the legal sector is well-positioned to capitalize on these developments. This thesis explores the feasibility of applying recent AI advancements, enhancing the performance of Large Language Models (LLMs) combined with the information retrieval capabilities of Retrieval-Augmented Generation (RAG), to resolve housing disputes in Finland.
The study provides an overview of chatbots or virtual assistants, associated technologies LLM and RAG, the opportunities chatbots offer for providing user-centric solutions, and the essential characteristics and challenges of applying chatbots in the legal domain. Furthermore, this study presents a case study on the development of an efficient system for legal information extraction using semantic Question and Answer (QnA) techniques applied to German case law documents. Additionally, this work sheds light on existing real-world chatbot solutions in the legal domain. As a result, this thesis delves into the current state, evolution, and future trajectories of the aforementioned technologies.
In this thesis, a prototype is developed using LLM technology optimized with RAG. Then, an experimental setup is designed to evaluate the performance of the RAG-optimized LLM against three non-optimized popular LLM-powered AI technologies, to assess the scope of RAG in improving LLM-powered chatbots. The experiment includes formulation of multiple prompts reflecting real-life housing dispute scenarios, including common user errors. The prototype has been developed on the MS Azure platform, integrating the LLM Azure OpenAI 3.5 turbo and Azure AI search for the RAG approach. The evaluation is based on testing the chatbots' comprehension and response generation capabilities.
The results from the experiment are evaluated by a human legal expert. The expert's analysis focuses on the accuracy and completeness of the responses generated by the models and the prototype. This evaluation helps identify the prototype’s RAG capabilities to retrieve information from the source documents as well as it identifies the challenges, limitations, and improvement criteria for adopting AI-powered virtual assistants in the legal field.
In conclusion, this thesis identifies the opportunities and unveils the gaps and intricacies in AI-powered chatbot’s capabilities to retrieve data from sources, to understand complex user scenarios, and to provide tailored responses aiming to provide meaningful guidance for users seeking solutions in the legal arena.
The study provides an overview of chatbots or virtual assistants, associated technologies LLM and RAG, the opportunities chatbots offer for providing user-centric solutions, and the essential characteristics and challenges of applying chatbots in the legal domain. Furthermore, this study presents a case study on the development of an efficient system for legal information extraction using semantic Question and Answer (QnA) techniques applied to German case law documents. Additionally, this work sheds light on existing real-world chatbot solutions in the legal domain. As a result, this thesis delves into the current state, evolution, and future trajectories of the aforementioned technologies.
In this thesis, a prototype is developed using LLM technology optimized with RAG. Then, an experimental setup is designed to evaluate the performance of the RAG-optimized LLM against three non-optimized popular LLM-powered AI technologies, to assess the scope of RAG in improving LLM-powered chatbots. The experiment includes formulation of multiple prompts reflecting real-life housing dispute scenarios, including common user errors. The prototype has been developed on the MS Azure platform, integrating the LLM Azure OpenAI 3.5 turbo and Azure AI search for the RAG approach. The evaluation is based on testing the chatbots' comprehension and response generation capabilities.
The results from the experiment are evaluated by a human legal expert. The expert's analysis focuses on the accuracy and completeness of the responses generated by the models and the prototype. This evaluation helps identify the prototype’s RAG capabilities to retrieve information from the source documents as well as it identifies the challenges, limitations, and improvement criteria for adopting AI-powered virtual assistants in the legal field.
In conclusion, this thesis identifies the opportunities and unveils the gaps and intricacies in AI-powered chatbot’s capabilities to retrieve data from sources, to understand complex user scenarios, and to provide tailored responses aiming to provide meaningful guidance for users seeking solutions in the legal arena.