AWS Data Engineer - AirFlow

Job Title: Airflow/AWS Data Engineer
Location: Manchester Area (3 days per week in the office)
Rate: Up to 400 per day inside IR35
Start Date: 03/11/2025
Contract Length: Until 31st December 2025
Job Type: Contract

An exciting opportunity has become available with one of our sector-leading financial services clients. They are seeking a talented AWS DevOps/Data Engineer to join their growing data engineering function. This role will play a key part in designing, deploying, and maintaining modern cloud infrastructure and data pipelines, with a focus on Airflow, AWS, and data platform automation.

• Deploy and manage cloud infrastructure across Astronomer Airflow and AccelData environments.
• Facilitate integration between vendor products and core systems, including data lakes, storage, and compute services.
• Establish and enforce best practices for cloud security, scalability, and performance.
• Configure and maintain vendor product deployments, ensuring reliability and optimized performance.
• Ensure high availability and fault tolerance for Airflow clusters.
• Implement and manage monitoring, alerting, and logging solutions for Airflow and related components.
• Perform upgrades, patches, and version management for platform components.
• Oversee capacity planning and resource optimization for Airflow workers and AWS resources.
• Manage integrations with source control systems (GitHub, GitLab) and CI/CD pipelines.
• Collaborate with AWS teams and internal stakeholders for pipeline scalability and optimization.
• Design and implement process improvements, including automation, data delivery optimization, and infrastructure re-design.
• Develop ETL pipelines and data workflows using AWS and SQL technologies.
• Partner with cross-functional teams (product, design, and leadership) to resolve technical issues and enhance platform capabilities.
• Build analytical tools and dashboards to leverage data pipelines for actionable business insights.

• Proven experience as an AWS DevOps Engineer or Data Engineer in complex cloud environments.
• Strong hands-on expertise with AWS services (EC2, S3, Lambda, RDS, IAM, CloudWatch, etc.).
• Demonstrated experience with Airflow (Astronomer) setup, orchestration, and optimization.
• Proficiency in infrastructure as code (IaC) tools such as Terraform or CloudFormation.
• Experience with CI/CD pipelines and tools like Jenkins, GitHub Actions, or GitLab CI.
• Solid understanding of containerization technologies (Docker, Kubernetes).
• Working knowledge of Python and SQL for automation and data pipeline development.
• Familiarity with monitoring and observability tools (Grafana, Prometheus, CloudWatch).
• Strong grasp of data architecture principles and ETL design patterns.
• Financial services or regulated industry experience (desirable).