Authors:
Aditya Rautaray, T. Shynu, R. Steffi, B. Sudha
Addresses:
Department of Cloud Solutions Security, CVS Healthcare, Ashburn, Virginia, United States of America. Department of Research and Development, Dhaanish Ahmed College of Engineering, Chennai, Tamil Nadu, India. Department of Electronics and Communication, Vins Christian College of Engineering, Nagercoil, Tamil Nadu, India. School of Science and Computer Studies, CMR University, Bengaluru, Karnataka, India.
Abstract:
This paper examines the problem as a specific instance of schema drift in high-velocity data streams and proposes a metadata-driven, adaptive ETL (Extract-Transform-Load) framework to address it. Traditional Data Integration tools come with these soft, brittle monitors, which fail down even if an upstream source changes its structure without prior notification. This weakness results in significant data loss and downtime. The architecture system uses event streaming (Apache Kafka), distributed processing (Apache Spark), and a dynamic Schema Registry abstraction to eliminate the need for a payload structure definition. The simulation has been performed on a synthetic dataset of 410 distinct IoT sensor telemetry cases, intentionally designed to simulate high-frequency changes in field types, column additions, and changes in granularity. A comparison with a typical static schema validation approach shows that the adaptive architecture yields substantially greater system robustness. The experiments reveal that the metadata-driven strategy achieves 99% system uptime during drift events, compared with 0% for static systems. The Results indicate that by changing the validation logic from hard-coded scripts to a dynamic metadata layer, organisations can build agile immunity against volatility in real-time data streams.
Keywords: Schema Drift; Adaptive ETL; Data Integration; Dynamic Schema Registry; Event Streaming; Distributed Processing; Apache Kafka; Apache Spark; System Uptime.
Received on: 03/05/2025, Revised on: 28/08/2025, Accepted on: 11/10/2025, Published on: 05/05/2026
DOI: 10.64091/ATICS.2026.000259
AVE Trends in Intelligent Computing Systems, 2026 Vol. 3 No. 2 , Pages: 95–103