Efficient processing and visualization of large-scale energy consumption time-series data
Eskina, Ekaterina (2025)
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi:amk-2025110827418
https://urn.fi/URN:NBN:fi:amk-2025110827418
Tiivistelmä
This bachelor's thesis presents the design and implementation of comprehensive business intelligence (BI) (Olavsrud & Fruhlinger, 2023) pipeline for processing and visualizing energy consumption time-series data. The primary objective was to build a scalable, cloud-ready solution capable of transforming raw consumption data into actionable insights through automated data modeling and dynamic visualization (Al-Aqrabi et al., 2015) The project was carried out in three iterative phases. The initial prototyping phase involved data preparation in PostgreSQL, including cleaning, normalization, and monthly partitioning to explore performance optimization techniques (The PostgreSQL Global Development Group, 2025-a) DuckDB was used in combination with Python (Pandas and Seaborn) to enable fast local prototyping and data exploration in a lightweight, in-memory environment (Raasveldt & Mühleisen, 2021) These tools allowed rapid development and refinement of the analytical logic before transitioning to the final cloud-native platform.
In the implementation phase, all validated data transformation logic was restructured and automated using Python scripts that interact with Azure SQL Database (Microsoft, 2025-a). This cloud-based architecture enables persistent storage, scalable querying, and seamless integration with Microsoft Power BI. Over 20 analytical views were created to support various use cases such as consumption trend analysis, anomaly detection, and site-level performance comparison. Interactive dashboards were developed in Power BI to visualize daily, hourly, and seasonal consumption patterns, as well as key data quality metrics.
The final BI pipeline successfully decouples data processing (via Python and Azure SQL) from visualization (Power BI), providing a robust, automated, and user-centric solution. Users are empowered to explore the dataset through intuitive filtering and drill-down functionalities, facilitating data-driven decision-making in energy monitoring and management.
In the implementation phase, all validated data transformation logic was restructured and automated using Python scripts that interact with Azure SQL Database (Microsoft, 2025-a). This cloud-based architecture enables persistent storage, scalable querying, and seamless integration with Microsoft Power BI. Over 20 analytical views were created to support various use cases such as consumption trend analysis, anomaly detection, and site-level performance comparison. Interactive dashboards were developed in Power BI to visualize daily, hourly, and seasonal consumption patterns, as well as key data quality metrics.
The final BI pipeline successfully decouples data processing (via Python and Azure SQL) from visualization (Power BI), providing a robust, automated, and user-centric solution. Users are empowered to explore the dataset through intuitive filtering and drill-down functionalities, facilitating data-driven decision-making in energy monitoring and management.