QT Sermo - A conversation chatbot using synthetic data, open-source solutions and integrated on a QTrobot via a Jetson Orin Nano

Abstract

Previous research suggests that users desire conversations with humanoid robots. Advances in Large Language Models (LLMs) now enable more effective conversation capabilities. This thesis develops a conversation system on the QTrobot platform with a Jetson Orin Nano, utilizing free and open-source software (FOSS) to prioritize user privacy and transparency. Automatic speech recognition (ASR), a local LLM, and a text-to-speech solution enables real-time conversations. Experimentation with fine-tuning the LLM, using synthetic data generated from multiple open models, showed promise but also revealed limitations. This thesis proposes an accuracy metric for evaluating synthetic data generation of LLMs based on insights from the experiments, building a framework for creating larger datasets to enhance other models in the future. The conversation system's modular design allows for deployment on various robot platforms; however, further testing is required to validate its performance across different robotics systems.