BREAKING NEW GROUND: THE EVOLUTION OF CLOUD-NATIVE DATA STREAMING PLATFORMS
DOI:
https://doi.org/10.34218/IJCET_16_01_270Keywords:
Cloud-native Streaming, Data Processing Architectures, Edge Computing Integration, Real-time Analytics, Serverless ComputingAbstract
This comprehensive article explores the evolution and impact of cloud-native streaming platforms in modern data engineering. The article examines how these platforms have transformed traditional data processing paradigms, offering enhanced scalability, reliability, and operational efficiency. It investigates key technical innovations including auto-scaling partitions, schema management, and stream lineage capabilities, while analyzing implementations at major technology companies. The article also delves into multi-cloud architectures and serverless processing capabilities, highlighting how these advances have revolutionized distributed data processing. The article evaluates real-world impacts across performance, operational efficiency, and business agility dimensions, concluding with an examination of future directions in machine learning integration, edge computing, and security enhancements for streaming platforms.
References
Theofanis P. Raptis, et al., "A Survey on Networked Data Streaming With Apache Kafka," in IEEE Access, vol. 11, pp. 84171-84183, 2023. [Online]. Available: https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=10213406
Mehdi Mohammadi, et al., "Deep Learning for IoT Big Data and Streaming Analytics: A Survey," IEEE Communications Surveys & Tutorials ( Volume: 20, Issue: 4, Fourth Quarter 2018). [Online]. Available: https://ieeexplore.ieee.org/abstract/document/8373692
Pedro Garcia Lopez, et al., "Edge-centric Computing: Vision and Challenges," Acm Sigcomm Computer Communication Review, Volume 45, Issue 5. [Online]. Available: https://dl.acm.org/doi/10.1145/2831347.2831354
Redowan Mahmud, et al., "Quality of Experience (QoE)-aware placement of applications in Fog computing environments," Journal of Parallel and Distributed Computing, 2019. [Online]. Available: https://dl.acm.org/doi/10.1016/j.jpdc.2018.03.004
Guoqiang Jerry Chen, et al., "Realtime Data Processing at Facebook," UC Berkeley Technical Report, 2016. [Online]. Available: https://pages.cs.wisc.edu/~shivaram/cs744-readings/Facebook-Streaming.pdf
Shadi A. Noghabi, et al., "Samza: stateful scalable stream processing at LinkedIn," Proceedings of the VLDB Endowment, Volume 10, Issue 12, 2017. [Online]. Available: https://dl.acm.org/doi/abs/10.14778/3137765.3137770
Scott Hendrickson, et al., "Serverless Computation with OpenLambda," in Proceedings of the 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 16), 2016, pp. 33-39. [Online]. Available: https://www.usenix.org/system/files/conference/hotcloud16/hotcloud16_hendrickson.pdf
Vikram Sreekanti, et al., "Cloudburst: stateful functions-as-a-service," Proceedings of the VLDB Endowment, Volume 13, Issue 12, 2020. [Online]. Available: https://dl.acm.org/doi/10.14778/3407790.3407836
Sanjeev Kulkarni, et al., , "Twitter Heron: Stream Processing at Scale," SIGMOD '15: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, 2015. [Online]. Available: https://dl.acm.org/doi/10.1145/2723372.2742788
[Paris Carbone, et al., "State Management in Apache Flink," Proceedings of the VLDB Endowment, vol. 10, no. 12, pp. 1718-1729, 2017. [Online]. Available: https://paper-notes.zhjwpku.com/assets/pdfs/state-management-in-apache-flink.pdf
Anton Gulenko, et al., "Evaluating machine learning algorithms for anomaly detection in clouds," IEEE International Conference on Big Data (Big Data), 2016. [Online]. Available: https://ieeexplore.ieee.org/document/7840917
Wazir Zada Khan, et al., "Edge computing: A survey," Future Generation Computer Systems, Volume 97, August 2019, Pages 219-235. [Online]. Available: https://www.sciencedirect.com/science/article/abs/pii/S0167739X18319903
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Himanshu Adhwaryu (Author)
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
