Building the Corporate Memory: A RAG-Based Chatbot for Contextual Retrieval from Unstructured Service Reports : Architecture and Inference Optimization on Apple Silicon Hardware
Le Tran, Vi (2025)
2025
All rights reserved. This publication is copyrighted. You may download, display and print it for Your own personal use. Commercial use is prohibited.
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi:amk-2025121636835
https://urn.fi/URN:NBN:fi:amk-2025121636835
Tiivistelmä
This thesis addresses the problem of knowledge loss in the maritime service industry caused by unstructured field service reports. The goal was to develop and evaluate a privacy-preserving, locally hosted chatbot that functions as an organizational memory system by making thousands of historical service reports searchable in natural language.
A Retrieval-Augmented Generation (RAG) architecture was implemented using the Mistral-Large-Instruct-2411 model running entirely on a Mac Studio M3 Ultra. Generic inference framework (Hugging Face Transformers) failed due to out-of-memory errors during generation despite successful model loading. Domain-Adaptive Pre-training (DAPT) was tested and rejected in favor of RAG due to excessive resource demands. The viable solution combined 4-bit quantization with the Apple-Silicon-native mlx-lm framework, reducing peak memory footprint by 70 % (from 250.9 GB to 74.65 GB) and increasing token generation speed from 2.6 to 7.9 tokens/second, thereby enabling interactive performance on a single workstation.
Evaluation shows high contextual accuracy, strict prompt adherence, and practical response times. The results confirm that powerful, fully local LLM systems are feasible for Kongsberg enterprise, offering a secure and cost-effective on-premises alternative to public cloud solutions for preserving and leveraging institutional knowledge.
This thesis was conducted in collaboration with Kongsberg Maritime Finland OY, Rauma, leveraging their resources and proprietary data. The necessary high-performance computing equipment was provided by RoboAI Academy.
A Retrieval-Augmented Generation (RAG) architecture was implemented using the Mistral-Large-Instruct-2411 model running entirely on a Mac Studio M3 Ultra. Generic inference framework (Hugging Face Transformers) failed due to out-of-memory errors during generation despite successful model loading. Domain-Adaptive Pre-training (DAPT) was tested and rejected in favor of RAG due to excessive resource demands. The viable solution combined 4-bit quantization with the Apple-Silicon-native mlx-lm framework, reducing peak memory footprint by 70 % (from 250.9 GB to 74.65 GB) and increasing token generation speed from 2.6 to 7.9 tokens/second, thereby enabling interactive performance on a single workstation.
Evaluation shows high contextual accuracy, strict prompt adherence, and practical response times. The results confirm that powerful, fully local LLM systems are feasible for Kongsberg enterprise, offering a secure and cost-effective on-premises alternative to public cloud solutions for preserving and leveraging institutional knowledge.
This thesis was conducted in collaboration with Kongsberg Maritime Finland OY, Rauma, leveraging their resources and proprietary data. The necessary high-performance computing equipment was provided by RoboAI Academy.