Robust Data Pipeline With NiFi, Spark and Lakehouse Architecture

Abstract

The rapid growth of data generated from sensors, transactions, mobile devices, and online platforms has created significant challenges for organizations seeking to transform raw data into actionable insights. As data volume and variety continue to expand, traditional ETL systems often suffer from limited scalability, high latency, and complex maintenance requirements. This has driven the need for modern, scalable, efficient, and fault tolerant data pipelines capable of reliably handling ingestion, transformation, and storage at scale. In contemporary data engineering, distributed frameworks and hybrid architectures are increasingly adopted to address these demands, yet challenges such as ingestion bottlenecks, uneven resource utilization during transformation, data skew, and the trade off between storage performance, reliability, and cost persist.

This thesis addresses these challenges by designing and implementing an end to end data pipeline that integrates Apache NiFi for high throughput ingestion, Apache Spark for distributed data transformation, and a Lakehouse architecture using Delta Lake for reliable and optimized storage. The New York City taxi trip dataset is used as a large scale real world case study, demonstrating the analytical potential of the proposed solution in domains such as transportation planning and business optimization. Performance is evaluated in terms of ingestion speed, transformation efficiency, and query optimization. The results show that NiFi provides reliable and efficient data flow management, Spark achieves substantial performance gains through distributed execution and partition tuning, and Delta Lake ensures ACID compliant storage while improving query performance. Together, these technologies illustrate how a modern, fault tolerant data pipeline can be effectively constructed, contributing practical insights and best practices for data engineering and smart city analytics.