Real-Time Data Integration and Processing : Advancing ETL Systems with Kafka and NoSQL
Vaze, Swarupa (2025)
2025
All rights reserved. This publication is copyrighted. You may download, display and print it for Your own personal use. Commercial use is prohibited.
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi:amk-2025122938961
https://urn.fi/URN:NBN:fi:amk-2025122938961
Tiivistelmä
Modern distributed systems increasingly demand flexible data models and efficient client-driven APIs. Traditional relational databases and RESTful interfaces often introduce limitations such as rigid schemas, complex migration, endpoint generation and inefficient data retrieval.
The purpose of this study was to investigate the feasibility of replacing a relational database with NoSQL database and REST APIs with GraphQL within an event-driven environment. Study recommends MongoDB and GraphQL as modern alternatives to resolve challenges in current company’s architecture with relational database and REST API. The theoretical background presents MongoDB, which is one of the leading NoSQL databases, and compares it with RDBMS to examine schema evolution and backward compatibility. With this GraphQL is explained where it compares with REST API to figure out data access efficiency with reduction in over-fetching and API evolution without versioning.
The result of this study was a proposed technical implementation, which contained Docker Compose incorporating MongoDB enterprise, Kafka, Kafbat UI for message monitoring, a Spring Boot Kafka consumer service for data persistence and a separate Spring Boot GraphQL client for data access.
The purpose of this study was to investigate the feasibility of replacing a relational database with NoSQL database and REST APIs with GraphQL within an event-driven environment. Study recommends MongoDB and GraphQL as modern alternatives to resolve challenges in current company’s architecture with relational database and REST API. The theoretical background presents MongoDB, which is one of the leading NoSQL databases, and compares it with RDBMS to examine schema evolution and backward compatibility. With this GraphQL is explained where it compares with REST API to figure out data access efficiency with reduction in over-fetching and API evolution without versioning.
The result of this study was a proposed technical implementation, which contained Docker Compose incorporating MongoDB enterprise, Kafka, Kafbat UI for message monitoring, a Spring Boot Kafka consumer service for data persistence and a separate Spring Boot GraphQL client for data access.