Implementation of log collection pipeline and visualization for a CI/CD environment
Vatto, Henri (2022)
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi:amk-2022121127894
https://urn.fi/URN:NBN:fi:amk-2022121127894
Tiivistelmä
The goal of the thesis was to create a log collection pipeline for a continuous integration environment. The unified logging layer would collect logs from servers running containerised software tests on embedded Linux devices. This data would be collected to a database and the data would be visualised on a web-based user interface. The project was conducted for GE Healthcare Finland, which is a medical technology company specialized in patient monitoring.
EFK stack is a collection of tools made for collecting, monitoring, and analysing log data. The stack consists of Elasticsearch for storing and indexing the data, Fluentd for aggregating the logs, and Kibana for visualization.
The EFK stack was implemented by containerizing all the parts of it for easier distribution and deployment. The logs were collected from systemd journal and sent to Fluentd through systemd journal upload. Elasticsearch was configured to run as a single node cluster with daily creation of indices. The Kibana dashboard was created with visualizations displaying information on the latest errors and warnings as well as general information of the test side logs.
The log collection pipeline was set up as planned and test servers were able to send their logs directly to it. As the pipeline performs adequately on the continuous integration testing environment, it can be extended to collect logs from the build systems as well.
EFK stack is a collection of tools made for collecting, monitoring, and analysing log data. The stack consists of Elasticsearch for storing and indexing the data, Fluentd for aggregating the logs, and Kibana for visualization.
The EFK stack was implemented by containerizing all the parts of it for easier distribution and deployment. The logs were collected from systemd journal and sent to Fluentd through systemd journal upload. Elasticsearch was configured to run as a single node cluster with daily creation of indices. The Kibana dashboard was created with visualizations displaying information on the latest errors and warnings as well as general information of the test side logs.
The log collection pipeline was set up as planned and test servers were able to send their logs directly to it. As the pipeline performs adequately on the continuous integration testing environment, it can be extended to collect logs from the build systems as well.