Improving the end-to-end testing of an MLOps platform

Abstract

With the emergence of large language models, there number of organisations adding machine learning to their operations has been increasing, particularly in fields such as autonomous systems, medical imaging, fraud detection, and logistics. However, deploying and maintaining production-ready machine learning models requires more than just computational resources — it also involves model monitoring, retraining, and managing data the datasets.

Machine Learning Operations (MLOps) is the equivalent of DevOps but instead of software development it focuses on machine learning. It addresses the previously mentioned challenges by providing tools and processes for managing the machine learning lifecycle. This includes deployment, orchestration, and performance tracking for the models.

There are several MLOps platforms available in the market to help organisations managing their machine learning workloads and models. Like with any software, when developing these platforms, it is important to do proper testing, especially when adding new features. This helps ensuring reliability of the product but enhances also its quality and the user experience.

This thesis focuses on improving the end-to-end testing capabilities for an MLOps platform called Valohai. The work included migrating the staging environment to Terraform-managed infrastructure and building realistic example projects that mimic customer use cases. As a result, it was possible to enhance internal debugging, help detect platform issues earlier, and support more reliable development and testing workflows.