ENHANCING DATA ANALYTICS WITH RETRIEVAL-AUGMENTED GENERATION WORKLOADS

Abstract

This article presents an innovative approach to enhancing data analytics by integrating Retrieval-Augmented Generation (RAG) workloads into traditional analytics pipelines. The article introduces a comprehensive framework that combines advanced retrieval mechanisms utilizing vector search and embedding models, generative pipelines with transformer-based architectures, and integration layers to address the growing challenges in processing and analyzing large-scale data. Through extensive testing across multiple data categories, including financial, healthcare, manufacturing, and customer behavior datasets, the system achieved a 91.7% accuracy in semantic search capabilities and a 93.4% improvement in pattern recognition using dense vector embeddings. The retrieval component, powered by a neural network architecture with 875 million parameters and trained on 623 million records, significantly improved precision and recall metrics. The proposed framework incorporates serverless computing architecture and automated pipeline management, enhancing performance across various operational metrics. The implementation considerations and guidelines offer a practical roadmap for organizations seeking to modernize their analytics infrastructure while maintaining security and scalability, with the RAG-enhanced system showing a 76.3% reduction in latency compared to traditional methods while processing datasets up to 1.2 petabytes in size.