Data Process Approach by Traditional and Cloud Services Methodologies

Abstract

The thesis examines the construction of data processing solutions for large data sets (over one million rows of data) utilising two distinct data processing pipeline architects. The purpose of this thesis was to develop and implement a process for extracting, transforming, and loading (ETL) raw data from a variety of different data sources into meaningful and useful information in a data warehouse based on enterprise resource planning (ERP) models. Action research was used in this thesis, as well as data mining and cleaning, as well as numerous methods of inspection for extraction. It compares two primary data storage services: traditional databases and cloud-based virtual computing instances with support for data orchestration. Testing techniques were used to analyse and certify the ETL specification. The research accomplished the primary goals of data integration and modelling by using a variety of technologies on a data instruction basis. The upstream departments will utilise the output of the process in conjunction with their own use to address various difficulties. This thesis provides deployment techniques, introduces libraries, and demonstrates how to use these libraries for data integration throughout the system's development. The deployment procedure was developed with the goal of resolving real-world difficulties. The result demonstrates both automated and non-automated ETL approaches.