Exploring Efficient Workflow Frameworks for Data Management
Petraityte, Ernesta (2024)
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi:amk-202404106175
https://urn.fi/URN:NBN:fi:amk-202404106175
Tiivistelmä
This thesis, titled "Exploring Efficient Workflow Frameworks for Data Management," focuses on improving data management strategies for the Sponsoring Consortium for Open Access Publishing in Particle Physics (SCOAP³). The objective is to evaluate how workflow management systems, particularly Apache Airflow, can enhance SCOAP³'s ability to manage a large volume of data effectively.
Structured in two main sections, the study first reviews relevant literature to set the theoretical groundwork for workflow management. It then conducts a comparative analysis of workflow management tools, with a detailed case study on Apache Airflow's application within the SCOAP³ project. The research methodology combines qualitative methods to assess the impact of these systems on data handling.
Findings from the case study indicate that integrating Apache Airflow leads to notable improvements in data workflow management, including enhanced follow-up on the state of data processing. These results suggest that workflow management systems play a critical role in streamlining data operations, especially in contexts dealing with extensive datasets.
The thesis concludes by linking these findings to existing literature and theories on workflow management. It confirms that while Apache Airflow enhances data management processes, its integration and optimization come with challenges. Future research is recommended to explore more adaptable workflow management solutions to further improve data management practices. This work provides a solid foundation for entities like SCOAP³ to leverage workflow management tools for more efficient data management.
Structured in two main sections, the study first reviews relevant literature to set the theoretical groundwork for workflow management. It then conducts a comparative analysis of workflow management tools, with a detailed case study on Apache Airflow's application within the SCOAP³ project. The research methodology combines qualitative methods to assess the impact of these systems on data handling.
Findings from the case study indicate that integrating Apache Airflow leads to notable improvements in data workflow management, including enhanced follow-up on the state of data processing. These results suggest that workflow management systems play a critical role in streamlining data operations, especially in contexts dealing with extensive datasets.
The thesis concludes by linking these findings to existing literature and theories on workflow management. It confirms that while Apache Airflow enhances data management processes, its integration and optimization come with challenges. Future research is recommended to explore more adaptable workflow management solutions to further improve data management practices. This work provides a solid foundation for entities like SCOAP³ to leverage workflow management tools for more efficient data management.