Sr Advanced Data Engineer

KEY RESPONSIBILITIES 

Data Engineering & AI Pipeline Development:

• Design and implement scalable data architectures to process high-volume IoT sensor data and telemetry streams, ensuring reliable data capture and processing for AI/ML workloads
• Build and maintain data pipelines for AI product lifecycle, including training data preparation, feature engineering, and inference data flows
• Develop and optimize RAG (Retrieval Augmented Generation) systems, including vector databases, embedding pipelines, and efficient retrieval mechanisms
• Lead the architecture and development of scalable data platforms on Databricks 
• Drive the integration of GenAI capabilities into data workflows and applications 
• Optimize data processing for performance, cost, and reliability at scale
• Create robust data integration solutions that combine industrial IoT data streams with enterprise data sources for AI model training and inference

DataOps:

• Implement DataOps practices to ensure continuous integration and delivery of data pipelines powering AI solutions
• Design and maintain automated testing frameworks for data quality, data drift detection, and AI model performance monitoring
• Create self-service data assets enabling data scientists and ML engineers to access and utilize data efficiently
• Design and maintain automated documentation for data lineage and AI model provenance

Collaboration & Innovation:

• Partner with ML engineers and data scientists to implement efficient data workflows for model training, fine-tuning, and deployment
• Mentor team members and provide technical leadership on complex data engineering challenges 
• Establish data engineering best practices, including modular code design and reusable frameworks 
• Drive projects to completion while working in an agile environment with evolving requirements in the rapidly changing AI landscape