Development of a Thesis Guidance Chatbot (PoC)
Veijanen, Jukka (2025)
2025
All rights reserved. This publication is copyrighted. You may download, display and print it for Your own personal use. Commercial use is prohibited.
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi:amk-2025120933965
https://urn.fi/URN:NBN:fi:amk-2025120933965
Tiivistelmä
The objective of the thesis was development and evaluation of the Proof of Concept (PoC) version of a RAG (Retrieval-Augmented Generation) chatbot implemented for thesis guidance. The thesis writing process often requires specialized knowledge and personal guidance, but the available guidance resources are limited. The need for an automatic, independent guidance system is obvious.
A locally operating RAG application was developed that is not dependent on third-party service providers. The application was implemented in Python, using the Langchain library for information retrieval and response generation, the Streamlit user interface, and ChromaDB for vector database management. Artificial intelligence models were integrated via the Ollama framework, using the large language models.
Since it was not possible to implement user testing, the functionality of the application was evaluated using the RAGAs tool, which allows for comparable measurement results. The chat functionality was tested with three different language models (Llama 3.2, Phi 3, Qwen 3), and the answers they generated were evaluated using three judge models (Codellama, Mistral, Qwen 2).
The developed RAG chatbot proved to be technically functional at the PoC level, able to handle thesis-related questions and provide guidance using locally installed language models. The testing results show that different combinations of language models and judge models produce varying results in terms of the quality of answers.
The study shows that independent RAG chatbots are technically feasible and offer a potential solution for automating thesis guidance. Further development requires more extensive user testing and optimization of the application according to real-world usage situations.
A locally operating RAG application was developed that is not dependent on third-party service providers. The application was implemented in Python, using the Langchain library for information retrieval and response generation, the Streamlit user interface, and ChromaDB for vector database management. Artificial intelligence models were integrated via the Ollama framework, using the large language models.
Since it was not possible to implement user testing, the functionality of the application was evaluated using the RAGAs tool, which allows for comparable measurement results. The chat functionality was tested with three different language models (Llama 3.2, Phi 3, Qwen 3), and the answers they generated were evaluated using three judge models (Codellama, Mistral, Qwen 2).
The developed RAG chatbot proved to be technically functional at the PoC level, able to handle thesis-related questions and provide guidance using locally installed language models. The testing results show that different combinations of language models and judge models produce varying results in terms of the quality of answers.
The study shows that independent RAG chatbots are technically feasible and offer a potential solution for automating thesis guidance. Further development requires more extensive user testing and optimization of the application according to real-world usage situations.