Design and development of an active monitoring solution for the integration of data pipelines at case company
Gonçalves Borrega, Facundo (2024)
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi:amk-2024120332564
https://urn.fi/URN:NBN:fi:amk-2024120332564
Tiivistelmä
The purpose of this thesis was to investigate data inconsistencies in a particular scope of data pipeline processes at the case company, Hipoges Iberia, in order to design, develop and deliver an active monitoring solution.
The theoretical framework of the thesis is based on the literature review of theories and previous research on key concepts on the subject, together with an analysis of current industry solutions, as to provide a strong base for the background and current perspectives on the topic.
This thesis used mixed research methodologies. Qualitative data was collected by interviewing key stakeholders affected by these data inconsistencies. Experimentation was used in order to gather quantitative data between development iterations and user-experience surveys for evaluating the solution.
The results of the survey indicated that the solution effectively detected data inconsistencies, it was easy to use and that it had good adaptability inside the case company. For future work, if the process is adapted for monitoring further scopes of data pipelines, components inside the solution need not to be dependent on the chosen frameworks, tools or coding languages.
The theoretical framework of the thesis is based on the literature review of theories and previous research on key concepts on the subject, together with an analysis of current industry solutions, as to provide a strong base for the background and current perspectives on the topic.
This thesis used mixed research methodologies. Qualitative data was collected by interviewing key stakeholders affected by these data inconsistencies. Experimentation was used in order to gather quantitative data between development iterations and user-experience surveys for evaluating the solution.
The results of the survey indicated that the solution effectively detected data inconsistencies, it was easy to use and that it had good adaptability inside the case company. For future work, if the process is adapted for monitoring further scopes of data pipelines, components inside the solution need not to be dependent on the chosen frameworks, tools or coding languages.